For those of us who are not practicing political scientists or do not yearn to have an article published in the American Political Science Review, “election forecasting” does not involve sophisticated statistical models or high-level quantitative analysis. That does not mean that we do not seek to identify variables that will be important in explaining—and ultimately projecting—the final outcome of an election. It does mean that we use qualitative judgments and general rules of thumb to base our analysis. In other words, our process cannot really be replicated.
Most of what I do—and the most important part of my work—does not involve “forecasting” elections. Unlike those political scientists who labor over their statistical models or the more popular prognosticators who try to make a career out of predicting the future, I spend far more of my time reporting on candidates and campaigns and attempting to put current political developments into context. But I realize that “predictions” have a particular appeal to most readers—whether they are predictions about politics, the stock market, the World Series, or the Academy Awards.
Many forecasters assign what I regard as overly specific chances for certain outcomes, but I have never done that and never will. The suggestion that there is a 77% chance someone will win a particular election, as compared to a 76% chance or a 78% chance, strikes me as silly. My approach is decidedly qualitative, and therefore I use an ordinal scale of nine categories—Safe Democrat, Safe Republican, Democrat Favored, Republican Favored, Lean Democrat, Lean Republican, Toss-Up/Tilt Democrat, Toss-Up/Tilt Republican, and Pure Toss-Up—to reflect my assessment of the relative vulnerability of seats.
Every election cycle, I start with the fundamentals: past state and district election results. I pour over election data to understand whether the behavior of voters in individual states and districts is changing, as well as how that change might affect future candidates and elections. As an election cycle starts, I begin evaluating both individual campaigns and the broad national political environment because both are important in trying to separate potential winners from likely losers. When everything else is equal, better candidates and better campaigns tend to beat worse candidates and worse campaigns. But everything else rarely is equal because broad national trends often have a significant effect at individual race level.
Because I cannot predict the future or how unanticipated events will change public attitudes toward the president, the political parties, or individual candidates, assessments made in April of an off-year may be very different from those made in April, July, or October of the election year. Given that, assessments made 18 (or 12 or even six) months before an election are not meant to be “predictions.” They simply reflect my judgment, at various points throughout an election cycle, of where races are headed and whether control of the chamber is “in play.”
EVALUATING CANDIDATES AND CAMPAIGNS
After weeding out hundreds of House districts and a good chunk of Senate contests where there is no chance of change in party control, my assessment of candidates’ prospects turns to their personal appeal and the quality of their campaigns, which includes dozens of factors, such as the manager and consulting team, fundraising, message, advertising, and get-out-the-vote efforts.
While campaign quality is important in evaluating a candidate’s prospects, I recognize that I have a very limited knowledge of individual campaigns. I watch TV ads and follow the media’s coverage of individual races, but I do not follow every campaign every day, and much of my information is second-hand. In-person interviews with candidates are invaluable because they allow me to make judgments about fundraising potential, communications skills, and ability to connect with voters. But a single meeting provides limited information, and many candidates learn on the job how to be good candidates and how to answer difficult questions. (A single meeting can be instructive, however. One meeting with a California Democratic congressional candidate last cycle who was running in a competitive district convinced me that the general election would not be competitive if that candidate won his party’s nomination.Footnote 1 He did, and it wasn’t.)
While I factor in my subjective evaluations about candidates’ appeal, I rely on a handful of quantifiable factors to evaluate candidates and their campaigns during the course of an election cycle: fundraising numbers, the size of television buys by campaigns and “outside” groups, and, most importantly, national and state or district polling that reflects the public’s mood and attitude toward particular candidates. I also rely on the assessment of others—both in Washington, DC, and in individual states and districts where competitive contests are occurring—who have more detailed information about the candidates and the campaigns. Of course, this constitutes nothing more than traditional “reporting,” as done by the likes of Dan Balz and the late David Broder, both of the Washington Post.
My understanding of the national environment is largely informed by public opinion surveys. Obviously, surveys in individual races are less valuable early in a cycle (when few voters know much about the candidates and the election’s context has yet to develop) and more important toward the end of an election cycle. In fact, during the final six weeks or so of an election, my assessments of races are based almost entirely on state-level and district-level survey data—some of it conducted by media outlets and released to the public, but much of it conducted by political insiders and never intended for public release. Early polls in House and Senate races are released primarily to help fundraising and establish a narrative about the competitiveness of a candidate. Because of that, they are far less useful than partisan polls that were conducted to offer strategic advice for candidates or campaign committees and therefore never released.
During the six weeks or so before Election Day, my assessments of individual races are only as good as the survey data that I see. Over the years, those data, which were provided to me by party operatives with the caveat that they could not be made public, have been very reliable.
At the end of an election cycle, just before an election, my colleague, Nathan Gonzales, and I “count” likely changes in party control to get a best estimate of net change. We try to force Pure Toss-Ups toward one party or the other, relying on polls but also on the general direction of an election cycle. Here is where “feel” comes into the equation in the final days and even weeks of an election cycle. In 2006, five days before Election Day, we moved the Virginia Senate race from Toss-Up to Lean Democratic (and wrote that Democrats would win control of the Senate) on the basis of our view that when there is a partisan political wave, most close contests fall to the party benefiting from the wave. (Democrat Jim Webb won the race, and Democrats did win the Senate.)
Four years later, we “counted” up all our competitive House races, pushing them to one or the other of the parties. Although we could only “count” a gain of about 55 seats for the Republicans, we decided that such a huge wave was developing that we probably were underestimating potential Republican gains. So, we increased our estimate of Republican gains to a historically high 55 to 65 seats, although we could not count individual district gains at that level. (Republicans won 63 seats that year.)
Our record is far from perfect. In 2012, we estimated the most likely Senate outcome as between no change and a Republican gain of three seats. Seeing no strong national partisan wave and believing that Democratic turnout levels would not come close to 2008, we assumed that toss-up contests would split roughly evenly between the parties, and the Republicans would hold onto a reliably Republican seat in North Dakota. In fact, most of the close races fell toward the Democrats, including the North Dakota seat, and Democrats gained two seats.
A POLLING HICCUP IN 2010 AND 2012, OR A POLLING CRISIS?
During the six weeks or so before Election Day, my assessments of individual races are only as good as the survey data that I see. Over the years, those data, which were provided to me by party operatives with the caveat that they could not be made public, have been very reliable. Recently, however, I have been growing less comfortable with the polling. At least initially, during the 2009–2010 cycle, Democratic pollsters underestimated the size of the Republican wave. Two years later, Republican pollsters seriously underestimated Democratic turnout, producing surveys that exaggerated Republican strength. Obviously, some national surveys were wrong, as well, with Gallup being the most obvious example.
Democrats had a much better handle of the state of the 2012 presidential race, and in the final few days of the 2012 election cycle, Republican strategists who were aware of the party’s House polls and had great confidence in them were quietly predicting small Republican gains. Instead, Democrats gained eight seats. Of course, not every Republican pollster missed the 2012 results, just as not every Democratic pollster got caught flat-footed in 2010. But over a period of two election cycles, each set of party pollsters underestimated the opposition’s strength during most of one cycle.
Given the changing demographic make-up of the country and the electorate, it is crucial for pollsters to figure out how to identify likely voters. Gallup acknowledged its failure to predict the outcome of the 2012 presidential race accurately, citing errors in its likely voter model (including a sample that was too white and overrepresented voters in regions of the country where Romney was stronger) and in its methodology (too few calls were made to cell phones, which produced an older sample) (Clement Reference Clement2013). Pollster Glen Bolger, one of the founding partners in the highly respected Republican polling firm Public Opinion Strategies, admitted many of the same errors less than a week after Election Day, and his partner, Bill McInturff, has explored the reasons why the traditional likely voter model failed (Bolger Reference Bolger2012; McInturff Reference McInturff2013). Both parties use sophisticated modeling, but increasingly high refusal rates and what appears to be a growing difference in survey results across various polls raises questions about the reliability of surveys, particularly in House and Senate races, where the sheer number of surveys is relatively small. (Obviously, this is not a problem in presidential polls, which seem to be conducted around the clock.)
CAN ANYONE SOLVE THE POLLING RIDDLE?
Some of polling’s recent problems could well reflect the industry’s uncertainties about Barack Obama’s unique appeal. Will younger voters and minorities continue to participate at the levels that they did in 2008 and 2012, or will the eventual end of the Obama Presidency cause voter participation to return to previous levels among some voting groups?
Survey researchers may well be able to better understand the shape of the electorate over the next few cycles, possibly establishing a “new normal” that will hold in most election cycles. Of course, forecasters who rely on public opinion surveys will still have to make distinctions among pollsters, because there is a clear difference in competence, skill, and intent. We would be in a much better position to understand the reasons for differing poll results in House and Senate races if all pollsters, not just those in academia and the media, would release more information about their samples, survey instruments, and overall methodologies. Unfortunately, this is not very likely to happen. Over the years, partisan pollsters are exceedingly hesitant about releasing too much of their methodology, seeing it as the heart of the business and of great proprietary value.
Polling will continue to be one way to understand what voters are thinking and how they are inclined to act on Election Day. Better reporting and more information about the polls will allow forecasters to understand why different organizations are getting different survey results and to weigh the results differently.
The decision by the National Election Poll, which conducted the 2012 national exit poll, to save money by surveying voters in only 31 states instead of all 50 certainly complicates things for those of us who try to understand what is happening at the state level. As reporters Jon Cohen and Scott Clement noted, the decision not to poll in 19 states, “will almost certainly limit post-election research for years to come” (Cohen and Clement Reference Cohen and Clement2012).
For some forecasters who rely on polling, there is an obvious strategy to deal with multiple polls: simply average all of the surveys to get the best possible idea where the race stands. That may be possible during a presidential race when there may be a handful of polls in a given week, but it is impractical in most House and Senate races, where poll data are more scarce. Moreover, I am not comfortable averaging all polls when I believe that some of them do not accurately reflect the standing of the contest. Because “garbage in” does produce “garbage out,” I prefer weeding out the likely garbage first—from pollsters I have little faith in and from polls that appear to be obvious outliers.
Because coming up with a number that purports to be the “exact” margin in a House or Senate race at a particular moment does not interest me, but only provides a general sense of where the race stands and where it might be headed, I do not jump through statistical hoops to deal with different poll results. I do, however, need to make a judgment about which polls are likely to be more accurate when surveys produce very different results, and more details about the samples, assumptions, and methodologies might well lead to a more informed conclusion about the value of particular surveys.
Polling will continue to be one way to understand what voters are thinking and how they are inclined to act on Election Day. Better reporting and more information about the polls will allow forecasters to understand why different organizations are getting different survey results and to weigh the results differently. But there will always be races that are too close to call, and there will always be surprises. That is part of what makes politics so interesting and why predicting election outcomes is such a popular pastime.
