Media priming is one of the best-known media effects. By drawing attention to certain national problems while ignoring others, television news programs help define the standards by which presidents are evaluated. However, despite its theoretical soundness and potential generalizability, the original study on media priming has rarely been replicated in a direct manner.
We position the present study as a direct replication following the dichotomy of direct vs. conceptual replication by Schmidt (Reference Schmidt2009), which is arguably the most widely shared taxonomy in contemporary psychology. Direct replication refers to the repetition of an original experimental procedure, whereas conceptual replication refers to the repetition of a test of earlier work using different methods (Schmidt Reference Schmidt2009). Direct replication is preferred to conceptual replication when the research purpose is to examine the validity of an original study because conceptual replication is uninformative, especially if it fails (Hendrick Reference Hendrick and Neuliep1991; Pashler and Harris Reference Pashler and Harris2012). Since the method used for conceptual replication is different from that used in the original study, a failed replication could be attributed to the false positive of the original study and/or to the methodological differences between the original and replication studies. In sum, direct replication is suited for fact confirmation and knowledge extension because it is more comparable with the methods of the original study and therefore its findings are more accumulable than is the case in conceptual replication.
Media priming was initially discovered by Iyengar et al. (Reference Iyengar, Kinder, Peters and Krosnick1984), but to the best of our knowledge no direct replications of the original study have been published in a peer-reviewed journal. Although a sizable number of conceptual replications and applied investigations using surveys have been published (see Roskos-Ewoldsen et al. (Reference Roskos-Ewoldsen, Klinger, Roskos-Ewoldsen, Preiss, Gayle, Burrell, Allen and Bryant2007) for a meta-analysis of the media priming effect, cogent alternative explanations of the phenomena, including projection and learning (Hart and Middleton Reference Hart and Middleton2014; Lenz Reference Lenz2009), have spurred a debate over the credibility of accessibility-based explanations of media priming (see also Miller and Krosnick Reference Miller and Krosnick2000).
Perhaps, the most comprehensive conceptual replications of the media priming effect in recent years were examined by Lenz (Reference Lenz2012). Using panel data to address the problem of reverse causation, Lenz (Reference Lenz2012) successfully replicated the media priming effect in performance issues such as those found in economics and those related to the personality of the president, but not in policy issues. It should be noted, however, that none of the tests by Lenz (Reference Lenz2012) are direct replications of those by Iyengar et al. (Reference Iyengar, Kinder, Peters and Krosnick1984). That is because the aim of Lenz (Reference Lenz2012) was to test whether voters follow or lead politicians on policy issues. Therefore, media priming is positioned as one of the mechanisms, including persuasion and learning that potentially realize issue voting. In contrast, the aim of the present study is to test the validity of media priming in the contemporary media environment.
The media environment has changed dramatically since the early 1980s, when media priming was first reported. The idea of media priming rests on the agenda-setting effect of TV, which is most plausible when people are situated in a homogeneous and low-choice media environment. However, the widespread use of cable TV and the Internet is making the assumption of a homogeneous audience less likely to hold, as a fragmented audience selectively tunes into a variety of content (Iyengar and Hahn Reference Iyengar and Hahn2009; Stroud Reference Stroud2011) and those with less interest in politics simply tune out from news (Prior Reference Prior2007). Furthermore, despite the potential generalizability of the media priming effect, which depends on accessibility heuristics, it has not been directly replicated in any context other than the US. All of these arguments underscore the importance of direct replication of the original media priming study in a contemporary media environment.
Based on these arguments, we conducted a direct replication of Experiment 1 by Iyengar et al. (Reference Iyengar, Kinder, Peters and Krosnick1984), with some changes. Specifically, we (a) collected data from Japanese undergraduates; (b) reduced the number of conditions to two; (c) used news coverage on the issue of relocating U.S. bases in Okinawa as the treatment; (d) measured issue-specific evaluations of the Japanese Prime Minister (PM) in the pre-treatment questionnaire; and (e) performed statistical analyses that are more appropriate for testing heterogeneity in the treatment effect. Consistent with the original experiment, we hypothesize that the treatment group that is exposed to a greater amount of coverage about the relocation of U.S. bases in Okinawa would attach greater importance than the control group to the relocation issue in evaluating PM Abe.
We did not find statistically significant evidence of media priming. Overall, the results suggest that the effects of media priming may be quite sensitive either to the media environment or to differences in the populations in which the effect has been examined.
METHOD
The design and pre-analysis plan of this study was preregistered in the Open Science Framework and the experiment was conducted in strict accordance with the preregistered plan.Footnote 1
Participants
Undergraduates at Kwansei Gakuin University were eligible to participate in the study, and 104 of them who were registered in the subject pool of the university completed the experiment between May 27 and June 11, 2015.Footnote 2 Thirty-one percent of the participants were male and the average age was 19. Fifty-seven percent of the participants were freshmen, 19% were sophomores, and 19% were juniors.Footnote 3 From May 22 to June 10, potential participants were initially contacted via email using the online subject pool management software Sona System, and those who agreed to participate in the experiment completed an online pre-treatment measure the day before coming to the laboratory. Participants were rewarded with a cash voucher worth 1,000 Japanese yen (approximately 8 U.S. dollars) on filling in the consent form at the laboratory.
Materials
Although the materials needed to be adjusted to the contemporary political context in Japan, this study made every effort to replicate essential details of Experiment 1 conducted by Iyengar et al. (Reference Iyengar, Kinder, Peters and Krosnick1984). A number of news stories were extracted from national commercial networks (JNN, NNN, FNN, and ANN) and two sets of news video clips were created for the treatment and control groups, both of which ran for approximately 40 minutes and contained 20 news stories.Footnote 4
The treatment consisted of news stories on the issue of relocating U.S. bases in Okinawa, Japan (hereafter referred to as the Okinawa issue; see Online Appendix A1 for a brief background of this issue). The video clip for the treatment group included eight news stories about the Okinawa issue (16 minutes long), whereas that for the control group made no reference to this issue at all. The stories on the Okinawa issue were distributed evenly throughout the video clip for the treatment group. All eight presentations on the Okinawa issue were filled out by stories on a variety of other contemporary events, such as a crash by a German airplane, a parade by a sexual minority group, and the successful launch of an information satellite. Neither video clip included more than one story on any of these other events. Video clips for the treatment and control group are available in the Open Science Framework.
Randomization
This experiment was designed to include two groups—the treatment and control groups—whereas the original experiment consisted of three groups: no exposure, intermediate exposure, and high exposure. We made this change to maximize the statistical power of the experiment. Moreover, in the data analysis of the original experiment, the intermediate- and high-exposure groups were collapsed to create the “some-exposure” group, and data were analyzed as if there were only two groups. Therefore, this change in the current study should not represent a serious deviation from the original study.
As many as three students were permitted to sign up for any single experimental session, and they were then randomly assigned to either the treatment or the control group by a random number generator (n treatment = 52, n control = 52). Table A1 in the Appendix shows that demographic and other characteristics were, as expected, similar between the treatment and control groups.
Procedure
All the participants completed an online pre-treatment measure before they arrived at the laboratory. When they arrived, they completed the consent form and received a cash voucher as a reward. They were then seated in front of a PC monitor in a separate room.
They were told that the purpose of the study was to investigate selective perception by examining how individual political values influenced their evaluation of television news, which is identical to the explanation provided in the original study. Participants were informed that they would be asked to view “typical” news stories for half an hour to test their selective perception. These would consist of a collection of daily news stories extracted from commercial national networks. Participants were also informed that after the news presentation, they would be required to complete two questionnaires, one assessing their political opinions and the other soliciting their reactions to the news stories. Then, the news video clip was shown. The experimenters observed the participants from an adjacent room using a video camera to ensure they were watching the video clip.
After watching the video clip, the participants underwent post-treatment measurement on the same PC on which they had watched the video clip. Finally, the participants were fully debriefed. The participants were asked to guess the true purpose of the experiment during the debriefing, but no one was able to guess correctly.
Pre-Treatment Measures
Evaluation of PM Abe's performance on the Okinawa issue
Evaluation of PM Abe's performance on the Okinawa issue was measured using a single item scored on a four-point scale ranging from 1 (very poor) to 4 (very good). Note that issue-specific evaluation of President Carter's performance was measured in post-treatment measurement in the original experiment. However, to avoid potential post-treatment bias (Hart and Middleton Reference Hart and Middleton2014; Lenz Reference Lenz2012), we measured this variable before the treatment.
Evaluations of PM Abe's performance on nuclear policy, the Trans-Pacific Partnership, pension reform, and protection against earthquakes
In addition to measuring the evaluation of PM Abe's performance on the Okinawa issue, we also measured evaluations of PM Abe's performance on nuclear policy, the Trans-Pacific Partnership, pension reform, and protection against earthquakes as covariates. Such issue-specific evaluations of PM Abe's performance were conducted to hide the true purpose of this experiment. By embedding evaluation of PM Abe's performance on the Okinawa issue into these covariates, we expected that participants would respond to different items without paying special attention to the Okinawa issue. Evaluations of PM Abe's performance on these issues were measured using a single item scored on a four-point scale ranging from 1 (very poor) to 4 (very good).
Details of measurements of other pre-treatment covariates including demographic variables, political interest, and ideology are described in Online Appendix A2.
Post-Treatment Measures
Overall evaluation of PM Abe's performance
Overall evaluation of PM Abe's performance was measured using a single item scored on a four-point scale ranging from 1 (very poor) to 4 (very good).
Evaluations of PM Abe's performance on the Okinawa issue, nuclear policy, the Trans-Pacific Partnership, pension reform, and protection against earthquakes
Immediately after measuring Overall evaluation of PM Abe's performance, issue-specific evaluations of PM Abe were measured again in exactly the same manner as the pre-treatment measure to differentiate between media priming and projection when interpreting the results (Hart and Middleton Reference Hart and Middleton2014; Lenz Reference Lenz2009).
Evaluation of PM Abe's competence
In accordance with the original study, evaluations of how well each of three adjectives (knowledgeable, smart, and weak) described PM Abe were measured using a four-point scale ranging from 1 (not well at all) to 4 (extremely well). These items were summed to form a competence index with ratings for “weak” being reversed (Cronbach's alpha = 0.61).
Evaluation of PM Abe's integrity
Evaluations of how well each of three adjectives (dishonest, power-hungry, and unstable) described PM Abe were measured using a four-point scale ranging from 1 (not well at all) to 4 (extremely well). These items were reversed and summed to form an integrity index (Cronbach's alpha = 0.42).
RESULTS
The media priming hypothesis predicts that the treatment group that watches stories about the Okinawa issue should attach greater importance to PM Abe's management of this issue than the control group does when evaluating his overall performance.
The original study first compared unstandardized regression coefficients indexing the effect of an issue-specific evaluation (an energy issue) in overall evaluations between a no-coverage group and a coverage group. Following the original study strictly, we estimated the unstandardized regression coefficients for evaluations of performance on the Okinawa issue in participants’ overall evaluations. Table 1 presents the results of our replication alongside those of the original study.Footnote 5 The four columns on the right-hand side of the table compare the coefficients of the treatment and control groups using a post-treatment measure of issue-specific evaluations in the same manner as the original study, while those in the middle of the table compare the estimated coefficients of a pre-treatment measure of issue-specific evaluations to address the potential post-treatment bias. To facilitate comparisons between the original study and the replication, we graphically illustrate the point estimates and their 95% confidence intervals (CI) in Figure 1.
Table 1 Impact of Issue-Specific Performance Evaluations on Summary Evaluations
Note: “Coefficients” are unstandardized regression coefficients.
Figure 1 Impact of Issue-Specific Performance Evaluations on Summary Evaluations.
Consistent with the original study, the coefficients are larger among the treatment group than among the control group with one exception: the effect of the post-treatment measure of issue-specific evaluation on the evaluation of competence (0.21 vs. 0.11). When the more credible pre-treatment measure of issue-specific evaluation is used, the treatment–control difference in overall performance evaluations (0.25 vs. 0.34) was less pronounced than that of competence evaluations (0.03 vs. 0.16), which does not replicate the original findings. Although it was in the same direction, the treatment–control difference of overall performance evaluations (0.25 vs. 0.34 for the pre-treatment and 0.39 vs. 0.44 for the post-treatment measure) was less pronounced than in the original study (0.10 vs. 0.27).
To test the statistical significance of these differences, we estimated OLS regressions that essentially replicated the model in the original study. Specifically, we estimated the following specification with three dependent variables: overall, competence, and integrity evaluations.Footnote 6
$$\begin{eqnarray*}
\textrm{Summary evaluation} &=& {\it b}_0 + {\it b}_1 \textrm{(Okinawa issue performance)} + {\it b}_2 \textrm{(Treatment)}\\[-4pt]
&& + \, {\it b}_3 \textrm{(Okinawa issue performance} \times \textrm{Treatment)} + {\it u},\\[-14pt]
\end{eqnarray*}$$
where Treatment is 1 for the treatment group and 0 for the control group. Note that we use a pre-treatment measure of Okinawa issue performance to avoid post-treatment bias, whereas the original study used a post-treatment measure of energy issue performance.Footnote 7
Table 2 presents the unstandardized regression coefficients of three regression models. Differences in issue weights and their 68% and 90% CIs are illustrated in Figure 2. Although they are in the expected direction, the interaction effects between Okinawa issue performance and treatment were not statistically significant across the three models, indicating that our replication of the media priming effect was unsuccessful. These results were not affected by the inclusion of sex, age, political interest, and ideology as covariates, although the magnitude of the coefficients is slightly greater when covariates are included (see Table OA4 in the Online Appendix).
Table 2 Regression Models Predicting the Summary Evaluations of PM Abe (OLS)
Standard errors in parentheses.
**p < 0.01, *p < 0.05.
Figure 2 Difference in Issue Weights between the Treatment and Control Groups.
It is interesting to note that the coefficients of the interaction term (i.e. Okinawa issue performance × Treatment) are larger than those of the original study: 0.09 vs. 0.07 for overall, 0.13 vs. 0.00 for competence, and 0.02 vs. –0.02 for integrity evaluations. However, none of the coefficients of the interaction term was distinguishable from zero in the replication, perhaps because of the larger standard errors.Footnote 8
Next, to differentiate between media priming and projection, we compared summary evaluations between the treatment and control groups, as the original study did. Previous studies (e.g. Lenz Reference Lenz2009) have pointed to possible alternative explanations of the original finding of a media priming effect. If overall evaluations were to be projected onto evaluations of specific performance on each issue, we would expect to observe changes in issue-specific evaluations after the treatment. Conversely, if the treatment changed the weight of issue-specific evaluations in overall evaluations, as proposed by the media priming hypothesis, we would not expect to observe significant treatment effects on evaluations because it is not change of evaluations but of accessibility that drives media priming. In line with the accessibility-based explanation of media priming, the original study did not find any statistically significant differences in post-treatment evaluations of President Carter.
The replication is presented in Table 3. Unlike the original study, the treatment effect on overall performance was statistically significant. That is, the treatment group's overall evaluation of PM Abe was less favorable than that of the control group. On the other hand, treatment effects of the evaluation of competence, integrity, and the post-treatment measure of his performance on the Okinawa issue were not statistically distinguishable from zero, which is consistent with the original study. This result is clearly at odds with the projection-based explanation, because overall evaluations are causally influenced by the treatment, while at the same time it does not offer clear-cut evidence of an accessibility-based priming effect. Combined with the null coefficients of the interaction term in Table 2, we must conclude that the direct replication of the original finding of media priming failed in this study.
Table 3 Treatment Effects on Post-Treatment Ratings
Note: Mean values of the original study are reversed so that larger values indicate better evaluations. All the treatment effects were insignificant in the original study.
DISCUSSION
Despite the fact that media priming is a classic example of mass media effects, and a considerable number of studies have provided empirical evidence supporting its existence, direct replications of the original study have been surprisingly scarce. In view of the rapid change of the media environment and the growing importance attached to replications in social science, the value of a direct replication of the original study of media priming is enormous. By means of a direct replication of the original study in contemporary Japan, this study tested the external validity and credibility of the original findings of a media priming effect. Our results indicate an unsuccessful replication. The rest of this section discusses potential reasons why the original media priming effect was not replicated.
First, although the present replication strictly followed the experimental procedure of the original study, the time and place were substantially different from the original. It is therefore hard to claim that the null result in the replication did not result from any of these situational differences between the two experiments. However, it is important to note that there cannot be an exact replication in social science because we are never able to use exactly the same participants in exactly the same social and political settings as in the original study (Schmidt Reference Schmidt2009). In light of the universality of availability heuristics on which the theory of the media priming effect is built, the media priming effect is implied to be quite foundational and generalizable across time and place. Furthermore, we carefully selected the treatment issue so that it resembles the one in the original study in essential ways. In particular, despite the substantial coverage in the news media, the salience of the Okinawa issue was not distinctively high compared with other issues. In addition, essential features of issues in regard to the media priming effect, such as uncertainty surrounding the responsibility of the political leader and the potential importance to the participants, were common between the Okinawa issue and the energy issue in the original study (see A1 in the Online Appendix). It is therefore unlikely that the ceiling effect and/or selection of the treatment issue caused our inability to replicate the media priming effect.
Second, recent developments in cultural psychology indicate that East Asians are more capable of embracing mutually contradictory ideas (e.g. Nisbett Reference Nisbett2003). If this is the case, even when evaluations of the PM's Okinawa issue performance are unfavorable, the correlation between the Okinawa issue and overall evaluations would be mitigated, so long as performance on other issues (e.g. economics) is evaluated favorably. Cross-national differences could have been influential in a more subtle way. For instance, the impact of issue-specific evaluation on the overall evaluation was larger in the replication than in the original study (see Table 1). This tendency was not limited to the Okinawa issue, but the impact of issue-specific evaluations was in general larger in the replication than in the original study.Footnote 9 Although it is beyond the scope of this study, cross-national differences in cultural and political variables, such as those that influence the baseline strength of association between issue-specific and overall evaluations, could have affected the reproducibility of the media priming effect.
Third, and most importantly, a dramatically different media environment may explain the unsuccessful replication. The original study was conducted well before the widespread use of cable TV and the Internet, and the participants were presumably embedded in a relatively homogeneous and low-choice media environment. However, in a media environment with an unprecedented amount of choices, modern news audiences consume enormously heterogeneous content through a variety of media outlets. If heterogeneity in terms of news consumption and political sophistication is greater among the participants in the replication than among those in the original study, a significant treatment effect might have been less likely to be detected because of the reduced efficiency of estimation. Furthermore, the rapid diffusion of mobile devices is making it more common to use a second screen for multitasking when watching TV; therefore, it might have been more difficult for contemporary undergraduates to focus on watching news for more than 30 minutes.
With the dramatic change of the media environment, there is pressure to update conventional theories of media effects (Bennett and Iyengar Reference Bennett and Iyengar2008). For a theory to be updated, it is essential to know the extent to which it holds in the modern media environment, which can only be discovered through well-crafted replications. In any case, a single replication study cannot provide conclusive evidence about the validity of the original study. The most reliable way to judge whether the media priming effect is real or not is to accumulate direct replications such as the present study that follow exactly the same procedure as the original study and subject them to a meta-analysis. Future studies of media effects should not only consider the consequences of the new media environment but also implement rigorous replications of extant findings.
SUPPLEMENTARY MATERIALS
The appendix is available online as supplementary material at https://doi.org/10.1017/XPS.2017.8
Appendix
Table A1 Descriptive Statistics and Balance Tests of Pre-Treatment Covariates
