Toward the end of the 2012 US presidential campaign, public poll aggregators converged on a forecast that President Obama would be reelected with victories in the same states he won in 2008, except Indiana and North Carolina, with Florida too close to call.Footnote 1 Meanwhile, most pundits and commentators took a contrary view: the race as a whole was no better than a toss-up. Some attacked the quantitative approach to election forecasting altogether (Nyhan Reference Nyhan2012). But the poll-based predictions turned out to be correct—even about Florida’s closest-in-the-nation election outcome. The ensuing media coverage took on an almost triumphal tone: “The Poll Quants Won the Election,” declared a headline in The Chronicle of Higher Education (Bartlett Reference Bartlett2012). The New Yorker called it a “Victory for the Pollsters and the Forecasters” (Cassidy Reference Cassidy2012). In The Economist (2012), it was a “March of the Nerds.”
How satisfied should political scientists—or, for that matter, anyone interested in rigorous approaches to politics and elections—actually be about this accomplishment? The answer depends on what we consider to be the purpose of election forecasting. Careful modeling of trial-heat survey data certainly beat the intuitions of political pundits (Greenfield Reference Greenfield2012). It also affirmed the credibility of public opinion surveys as a method for learning about voter preferences (Blumenthal Reference Blumenthal2012). On the other hand, forecasts issued just on the eve of the election arrived too late to help reporters, news consumers, or researchers understand how and why voters’ preferences had responded to the campaign. Nor were they much use to candidates and party strategists who needed to decide how to allocate resources for organizing, advertising, and voter mobilization much earlier in the election season.
Accurate election forecasts have considerable value to political observers as well as scholars and practitioners. To maximize these benefits, however, forecasting models need to generate predictions early in the race—preferably, three to four months before Election Day. They should also be accompanied by informative statements of uncertainty. Initial forecasts can be updated with newer information as the election nears; for example, from the results of public opinion polls. In addition, forecasting models should strive to generate predictions at the same level (state or district) as the election of interest. National forecasts that aggregate lower-level results are less useful than district-by-district predictions. And although most current forecasting models focus on the US presidential race, researchers should be exploring how to extend existing methods to other types of races and to contests outside the United States.
By any of these standards, forecasting elections is still far from a “solved” problem. In the lead-up to the 2012 election, Campbell (Reference Campbell2012) catalogued twelve regression-based forecasting models that extrapolated from past presidential election outcomes to predict the upcoming vote. Only seven correctly foresaw an Obama victory. Among the incorrect forecasts, two gave Obama as little as a 10% or 11% chance of winning.Footnote 2 Nine models only offered predictions of national, rather than state, vote outcomes. None of the twelve models provided any mechanism for correcting inaccurate estimates closer to Election Day. The performance of presidential election forecasting models was somewhat better in 2008 (when Obama won by a larger margin), but completely missed the narrow victory of George W. Bush in 2000 (Campbell Reference Campbell2001; Reference Campbell2008). Forecasting models for US congressional and gubernatorial elections have faced similar challenges (e.g., Klarner Reference Klarner2009; Lewis-Beck and Rice Reference Lewis-Beck and Rice1984; Peltzman Reference Peltzman1987).
The issue is not that the theories of voter behavior or campaign dynamics underlying any of these models are badly flawed or even terribly incomplete. Political scientists have long been aware that, broadly speaking, incumbents fare better when the economy is improving and when they (or their party) are viewed more positively in the electorate (e.g. Lewis-Beck and Stegmaier Reference Lewis-Beck and Stegmaier2000; Nannestad and Paldam Reference Nannestad and Paldam1994). Factors related to candidates’ incumbency status, the prevalence of identity-based voting, and variation in electoral rules and governing institutions may attenuate or amplify the effects of these core structural variables (Duch and Stevenson, Reference Duch and Stevenson2008).
The difficulty, instead, is in operationalizing and measuring each of these factors and figuring out how much each one contributes to the forecast. For example, should an “improving economy” be interpreted as increasing gross domestic product, rising household income, falling unemployment rates, or something else? If more than one variable is used to predict the election outcome, then which ones, and in what combination? Political science theories are rarely specific enough to say, but the small size of most election datasets precludes an empirical solution to the problem. Without more data, selecting variables based on the strength of their observed association with historical election outcomes runs the risk of over-fitting the forecasting model and degrading its ability to predict out-of-sample. Ultimately, a substantial portion of any election outcome is going to be random and unpredictable, no matter how good election science becomes.Footnote 3 Elections are complex, stochastic events. This “noisiness” not only limits the accuracy of forecasts from even well-specified models but also makes it nearly impossible to adjudicate between alternative model specifications. Different models can, and will, generate highly divergent election forecasts even if they are equally justifiable from a theoretical standpoint.Footnote 4
Fundamentals-based election forecasting is running into the limits of what additional theory is going to contribute. The greatest impediment to the development of better election forecasting models is not a lack of theory; it is a lack of data. Short of waiting another 50 years until the signal-to-noise ratio in election data tilts somewhat more favorably in our direction (and hoping that in the meantime, election conditions remain consistent), forecasters must find new and better sources of data to inform their predictions. Advancements in election forecasting will come from researchers who identify—or gather—these data, devise and test theories about how the data relate to the election outcomes we care about, and build the modeling technology to produce forecasts in a timely manner.
Fundamentals-based election forecasting is running into the limits of what additional theory is going to contribute. The greatest impediment to the development of better election forecasting models is not a lack of theory; it is a lack of data.
MORE DATA: POLLS... OR WHAT?
One of the most promising new sources of data over the last few years has been the large number of trial-heat opinion polls that are conducted—and released publicly—by survey research firms, media organizations, advocacy groups, and others. During the 2012 US presidential campaign, the results of more than 1,200 state-level polls were published, representing more than one million Americans. In 2008, more than 1,700 state polls were made available. Hundreds more polls were conducted at the national level.
For forecasters and analysts, this increase has represented a tremendous breakthrough. By applying the basic principle that sampling error in the individual polls can be cancelled out by averaging the results of concurrent surveys, poll aggregators have been able to estimate smoothed trends in state- and national-level voter preferences during the campaign,Footnote 5 and, in some cases, project these trend lines forward to Election Day (e.g., Linzer Reference Linzer2013; Silver Reference Silver2012). The projections combine polling data with information from historical models, applying the principle that by Election Day voter preferences will “revert” toward the outcome implied by the election fundamentals (Kaplan, Park, and Gelman 2012). Another variant of this procedure shrinks the forecasts toward a tied result, which helps prevent overconfidence.
The long-term availability of public trial-heat polling data can not be taken for granted, however. Public polling is completely decentralized and growing more expensive. Fewer state polls were conducted in 2012 than in 2008. At least 20 polls per state are needed in the final three months of the campaign to generate consistently accurate forecasts, but most states in 2012 had nowhere near that many.Footnote 6 In addition, within the polling industry there are complaints that aggregators are exploiting pollsters’ data. After the 2012 election Gallup editor-in-chief Frank Newport remarked that survey aggregators “don’t exist without people who are out there actually doing polls,” and that aggregation threatens to dissuade survey organizations from gathering these data in the first place (Marketplace 2012).
Beyond the polls, how can forecasters supplement historical election data? Some of the more exploratory recent efforts include data from prediction markets, which combine the judgments of large numbers of individuals about the likelihood of a future event; social media data; Internet usage or search data; or patterns of online political activism. We are only beginning to study how these data are generated, how they relate to election outcomes, and how they can be integrated into election forecasts. The potential value in these new types of information is not primarily in testing theories of voter behavior (although accurate forecasts can reflect positively on the underlying theory), but rather in measuring voter preferences and forecasting outcomes in ways that could not be done before.
IMPROVING TECHNOLOGY TO DO BETTER SCIENCE
There are many questions in the study of voting, campaigns, public opinion, and elections that political scientists could investigate more thoroughly with additional behavioral data and improved campaign monitoring and forecasting technology: Why do people vote the way they do? Why do people turn out to vote at all? When do people decide who they will vote for, and why? What types of voters change their minds during a campaign? What effect do campaigns have on the election outcome? The technology might be a statistical forecasting model, a measurement technique, or any systematic approach to extracting quantitative information from noisy election data. The aim is to put theories from political science to practical use.
A burst of innovation along these lines has recently occurred in the professional political arena (Issenberg Reference Issenberg2010, Reference Issenberg2013). An empirical, experimental approach to understanding (and, sometimes, guiding) political behavior is at the core of the research being done at organizations such as the progressive Analyst Institute, or inside the analytics department of the 2012 Obama presidential campaign (Issenberg Reference Issenberg2012; Scherer, Reference Scherer2012). The Republican National Committee recently hired its first-ever chief technology officer to revamp the party’s digital campaign infrastructure (Ward Reference Ward2013). A host of other data-driven consulting firms have sprung up since the 2012 elections on both the Democratic and Republican sides (Wilner Reference Wilner2013a).
While the motivations and objectives of academic political scientists clearly differ from those of partisan political strategists, the data analytic methods and approaches that each use overlap. Political science can be a model for how this research and technological development proceeds. The academic field, unlike the world of consultants, campaigns, and media pundits, has norms favoring transparency and replicability. Statistical models entail assumptions; a lack of transparency prevents us from evaluating those assumptions in any meaningful or constructive way, determining how sensitive the conclusions may be to particular methodological choices, or learning more about the relationships an analyst claims to be seeing in the data. There is less reason to trust research that is overly secretive or “proprietary.” Especially in politics, an open research model not only advances the science; it helps avoid charges of manipulation or bias (most likely from whichever side a forecaster is predicting to lose). Scholars who contribute to public discourse around campaigns and elections also offer a credible counterpoint to the often exaggerated or misinformed claims of political pundits.
Looking ahead, proving the value of political science to electoral politics might promote opportunities for similar contributions in other areas. Debates over international security, inequality, criminal justice, education, health, and the environment—to name only a few—can all benefit by drawing on empirical scientific evidence. The debate over whether sophisticated quantitative political research can be “relevant” to contemporary politics should be settled.
