Scoring lawmakers based upon the votes they cast while serving in Congress is both commonplace and politically consequential. However, scoring legislators' voting records is not without its problems—even when performed by organizations without a specific policy agenda. A telling illustration of the impact that these scores have on political debate recently arose in the 2004 Democratic presidential primaries.
On February 29, 2004, the non-partisan National Journal released its analysis of congressional voting for 2003. Of particular interest was the finding that Senator John Kerry—the Democratic nominee for president—was identified as the most liberal senator in 2003 based upon the analysis of 62 key votes. This finding was widely publicized and became the subject of a Democratic presidential debate held the very next night in New York City.
Despite being conducted by a nonpartisan source, we show that the ratings (and their subsequent influence) are misleading in at least two respects. First, ignoring the uncertainty associated with the scores is politically consequential; the conclusion of the National Journal (publicized by the press) that Kerry is the most liberal senator in 2003 is doubtful if the votes Kerry missed while campaigning for the Democratic nomination are properly accounted for.
Second, the voting scores were not placed in a proper political context. What made the ratings newsworthy were the scores of the Democratic presidential candidates. But a fairer picture emerges when we consider President Bush's position on issues considered by the Senate. Including President Bush in the analysis significantly changes the political interpretation of the data: Kerry appears to be as liberal as Bush is conservative for the votes that the National Journal analyzed in 2003. In short, Kerry's apparent liberalism is only half of the story; the other piece of news is the apparent conservatism of President Bush. Given that we can reasonably expect charges of extremism to be part of the cut and thrust of this year's election campaign, we offer the following analysis as a modest corrective.
Roll Calls and the 2004 Presidential Election (So Far)
Using roll call votes to track lawmakers' behavior is prevalent both in the political arena and in academia. Politically motivated groups such as the Americans for Democratic Action, the American Conservative Union, the Sierra Club, the National Federation of Independent Business, the U.S. Chamber of Commerce, the National Rifle Association, the National Right to Life organization, NARAL, and the AFL-CIO all routinely score legislators according to whether legislators vote in accordance with their respective policy agendas. These resulting scores are used not only to mobilize members and target incumbents in elections (e.g., the Americans for Democratic Action “Zeros and Heroes” list), but the groups also seek to influence lawmakers' votes by publicizing which votes are going to be scored ahead of time. As CQ Weekly reports: “interest groups—and members of Congress themselves—use legislative scorecards for much more than reflecting an assessment of each lawmaker's record or mobilizing supporters at the grassroots level at election time. Scorecards, interest groups and lawmakers agree, have become an effective tool of the lobbying trade” (Cochran 2003, 924).
Non-partisan organizations and publications such as Project Vote Smart, the National Journal, and CQ Weekly also routinely and independently score legislators. Unlike the scores produced by partisan organizations (which are intended to serve political purposes), the scores of nonpartisan publications aim to present an objective assessment of lawmakers' voting behavior. While the voting scores given by partisan groups reveal the extent to which a lawmaker agrees with the groups' policy agenda, the scores of non-partisan groups rank a legislator within the political spectrum between two extremes: liberal or conservative. In so doing they provide an important public service. For instance, an interested citizen might well rely on the nonpartisan scores as reasonably unbiased summaries of the ideological positions of their congressional representatives, perhaps with a view to assessing for whom they should vote in the next election.
The scores of ostensibly neutral arbiters like National Journal shape political discourse in less passive ways. Precisely because the source is neutral, noteworthy features of the National Journal scores attract the attention of the press and the campaigns. In particular, candidates seeking to define one another as extreme and “out-of-step” with voters may use the scores to validate their claims. It is well-known that a “problem for members of Congress seeking the presidency is their extensive voting records, which can—and usually are—deciphered, deconstructed and even distorted by opponents looking to given those positions a negative spin” (Martinez 2004, 458). Evidence from impartial sources often provides the basis for such claims.
The Democratic presidential nominee, Senator John Kerry (MA), is susceptible on this score, if only because his voting history is so long: 6,310 recorded votes cast over 20 years of service in the Senate. Since the candidacy of Lyndon B. Johnson, only two other presidential candidates have had lengthier roll call records (and therefore recorded positions on national issues): Bob Dole and Gerald Ford. As CQ Weekly reports: “Republicans are combing through the 6,310 votes Kerry has cast in the Senate to date, looking for this or that position as an indication that the likely Democratic challenger to President Bush is out of step with mainstream America” (Martinez 2004, 458).
Precisely this opportunity presented itself when the non-partisan National Journal issued its ratings of lawmakers for 2003 in January of 2004. The ratings revealed that Kerry was the most “liberal” senator in 2003. In interpreting this result, the National Journal wrote, “To be sure, Kerry's ranking as the No. 1 Senate liberal in 2003—and his earning of similar honors three times during his first term, from 1985 to 1990—will probably have opposition researchers licking their chops” (Cohen 2004, 618).
The national impact was immediate. On February 28, 2004—the day after the scores were released—Kerry's ranking was reported in the Boston Herald (“Liberal Label Sticks; GOP Jumps on Kerry Over Senate Ranking”), the New York Post (“Survey: Kerry A Flaming Liberal”) and the Washington Times (“List Says Kerry Top Senate Liberal”). Kerry's ranking also surfaced during the Democratic presidential debate sponsored by CBS and the New York Times held on February 29. Elizabeth Bumiller, a reporter for the New York Times, questioned Kerry about his rating (FDCH 2004):
BUMILLER: The National Journal, a respected, non-ideological publication covering Congress…has just rated you, Senator Kerry…the most liberal senator in the Senate. How can you hope to win with this kind of characterization, in this climate?
KERRY: Because it's a laughable characterization. It's absolutely the most ridiculous thing I've ever seen in my life.
BUMILLER: Are you a liberal?
KERRY: I mean, look, labels are so silly in American politics. I was one of the first Democrats in the United States Senate in 1985 to join with Fritz Hollings in deficit reduction. Now, does that make me a conservative? I fought to put 100,000 police officers on the streets of America. Am I a conservative?
BUMILLER: But, Senator Kerry, the question is…
KERRY: Do you know what they measured in that? First of all, they measured 62 votes. I voted 37 times; 25 votes they didn't even count because I wasn't there to vote for them. Secondly, they counted my voting against the Medicare bill, which is a terrible bill for seniors in America, they called that being liberal. Lots of conservatives voted against that. In addition, they counted my voting against George Bush's tax cut that we can't afford. I thought it was fiscally conservative to vote against George Bush's tax cut. They call it liberal.
BUMILLER: Is this a helpful characterization in this campaign?
KERRY: I think it's the silliest thing I've ever heard.
In addition to Kerry's dismissal of the National Journal scores, Senator John Edwards responded: “I don't think anybody in America cares about what some inside-Washington publication says about your ideology.” Perhaps. But there is no doubt that the charges as to the “extremism” of both Kerry and Bush will play a critical role in the campaign for president this year.
Since roll call scores permit both the press and partisan groups to characterize (or validate characterizations of) opposition candidates, an understanding of roll call scores is essential in determining the extent to which they should influence political discourse. For example, how much weight should be given to the ratings of the National Journal (using a methodology developed by Bill Schneider—a political scientist and senior political analyst for CNN and frequent commentator on contemporary politics) in light of Kerry's criticisms; is Kerry's reaction mere political spin or does it speak to a more serious criticism of the scoring procedure?
Also, how does President Bush rate relative to Kerry? Even if Kerry is among the most liberal senators, the political relevance of this characterization depends upon the extremism of his opponent. For instance, the choice between a liberal and a moderate is much different than the choice between a liberal and a conservative.
Identifying the Most Liberal Senator(s)
The National Journal scores are based on 62 key roll calls from the Senate, and 73 from the House. In reporting its methodology, the National Journal notes:
The ratings rank members of Congress on how they vote relative to each other on a conservative-to-liberal scale in each chamber. The scores are based on lawmakers' votes in three areas: economic issues, social issues, and foreign policy. The scores are determined by a computer-assisted calculation that ranks members from one end of the ideological spectrum to the other, based on key votes…selected by National Journal reporters and editors (Cohen 2004, 615).
In addition to publishing the results in a February 28 cover story entitled “How They Measured Up,” the scores are posted on the National Journal web site. The most newsworthy result in the rankings is that John Kerry is reported to be more liberal than 97% of the Senate, and more conservative than 4% of the Senate. No other Senator is ranked “more liberal” in the National Journal rankings. John Edwards (NC) is ranked “second most liberal” by this measure; Edwards is estimated by National Journal to be more liberal than 95% of the Senate, and more conservative than 6% of the Senate.
One problem with the rankings is that there are no confidence intervals for the reported scores. It is well-known that the results from public opinion polls are inexact measures of true public opinion and that the error associated with the results must be acknowledged. The same is also true for roll call scores. In fact, the uncertainty that accompanies a voting score (or ought to) is most easily explained using analogies from the realm of public opinion polling. Uncertainty results from using a small number of roll call votes, just as a small sample size induces uncertainty in the results of an opinion poll. There is also the question of validity: the legislator's decision on any analyzed vote may well have been influenced by party pressure, presidential pressure, and/or lobbying by interested groups, and is not a perfect reflection of the legislator's ideology (analogously, survey questions are imperfect, and do not tap respondents' attitudes the way we think they might). Finally, the lawmaker may have missed some votes (i.e., surveys are subject to bias from non-response).
These sources of error have important political consequences for the proper interpretation of Kerry's voting record (indeed, any voting record) and the 2003 National Journal scores. As Kerry noted in the CBS/New York Times debate, as a result of his presidential campaign, he (and the other Democratic presidential candidates) missed a sizable fraction of the votes that the National Journal analyzed. For the 62 key roll calls in the Senate in 2003, the average abstention rate is 2.9%—with 56 senators voting on every analyzed vote and another 23 senators voting on all but one. Interpreting the announced positions of the Bush administration on the 62 votes (as reported in CQ Weekly) as evidence of how President Bush would have voted were he in the Senate allows us to compare the percentage of recorded votes by Bush and the Democratic presidential candidates. Table 1 summarizes the findings.
As is immediately evident from Table 1, the Bush administration and the presidential candidates took positions on only some of the analyzed votes. In fact, the two presidential candidates—Bush and Kerry—announced positions or voted only 40% of the time. Kerry was right to suggest in the CBS/New York Times primary debate that his absenteeism is consequential—there is far less data with which to estimate his voting score (and those of his fellow presidential candidates) than for the typical senator. Using fewer votes (i.e., smaller sample size) to score Kerry means that Kerry's score will have a larger confidence interval than those of other candidates. Any conclusion about Kerry's voting score and his relative rank ordering in the Senate should be sensitive to this important feature of the roll call data.
To demonstrate both the validity of Kerry's reaction and the importance of reporting confidence intervals for vote scores we first replicate the analysis of the National Journal using the method outlined in Clinton, Jackman, and Rivers (2004). The intuition underlying statistical models of legislative voting such as that employed by the National Journal and Clinton, Jackman, and Rivers is that each roll call presents each legislator with a choice between a “yea” and a “nay” position. Legislators are presumed to vote for the position most similar to their own ideal policy position/outcome. As a matter of practice, a legislator's voting record probably reflects a number of different influences, including: personal ideology, the ideology of the legislator's constituency, lobbying by interest groups, and pressure from party leaders. Without considerably more data than is available here, the effects of each these plausible sources of influence can not be ascertained.1
On this point, and for various attempts to untangle the effects of ideology/constituency/party, see Jackson and Kingdon (1992), Levitt (1996), Snyder and Groseclose (2000), Erikson and Wright (2001), McCarty, Poole, and Rosenthal (2001), Krehbiel (2003), and Clinton, Jackman, and Rivers (2004).
Accordingly, our voting scores should not be literally treated as a measure of a senator's personal ideology, but rather as a mix of these possible influences on roll call voting, and, in any event, as a useful summary of the ideological content of a senator's voting record. We normalize the scores such that senators with more liberal voting histories have lower scores, and senators with more conservative voting histories have higher scores.
We present our results in a series of graphs. Figure 1 shows our estimated voting scores for the 100 senators (solid points) along with a 95% confidence interval for each voting score (horizontal bars). Negative scores are associated with more liberal preferred positions and positive scores represent more conservative preferred positions. Solid points indicate Democratic senators, generally to the left of Republican Senators (open points)—the notable exception being Zell Miller (GA).
Point Estimates and 95% Confidence Intervals using 62 National Journal Key Votes
We see immediately that some senators' voting scores are estimated with considerably wider confidence intervals than other senators. This is a product of two factors: (1) the relative lack of data resulting from absenteeism—all else equal, the less data we have the wider the confidence intervals—and (2) extremism- all else equal, as ideological extremism increases the confidence interval widens because the voting record becomes less informative (e.g., while the data suggest George Allen (VA) is relatively conservative, since we seldom observe Senator Allen voting for the more liberal position on any given roll call we do not know precisely how conservative he is).2
The analogy from an educational setting is the student who gets every question asked of them correct: until the student is asked questions that they answer incorrectly, all we know is that the student is relatively smart, but we lack a precise estimate of how smart the student is in an absolute sense. Indeed, this is one of the motivations behind adaptive testing in standardized tests; i.e., tailoring the difficulty of items to the ability of test-takers, as revealed by their responses to items answered earlier in the test.
Both features appear to influence Senator Kerry's score; our best guess is that Kerry is quite liberal, but it is difficult to precisely state “how liberal” given the combination of a onesided voting record and the prevalence of missing data.
Figure 2 shows the ranks of the estimated voting scores (and associated 95% confidence intervals) for the Senate. Senators have been arranged according to their estimated ideal points. By definition, the ranks and their confidence intervals must lie within 1 and 100 inclusive. Again, solid (open) points indicate the most likely rank for the indicated Democratic (Republican) senator, while the horizontal bars indicate 95% confidence intervals. Given that the voting scores graphed in Figure 1 are estimated with uncertainty, so too will be any rank-ordering based on these scores. Accordingly, we estimate the probability that a given senator is the most liberal senator (has a rank of 1): only three Senators have any appreciable chance of being the most liberal Senator, namely Kerry (0.38 probability), Senator Paul Sarbanes of Maryland (0.28), and Senator Jack Reed of Rhode Island (0.24).
Rank Ordering and 95% Confidence Intervals using 62 National Journal Key Votes
We can also compare any two senators to assess the probability that they are statistically distinguishable. For instance, the probability that Senator Edwards has a voting score more conservative (greater) than Senator Kerry's is .90, short of the traditional 95% standard used in social-scientific research but moderately strong evidence nonetheless. The probability that Senator Joseph Lieberman (CT) has a more conservative voting record than Senator Kerry is an overwhelming .998 (which is the same probability that Lieberman's voting record is more conservative than Edwards').
Note also that Kerry's voting score is indistinguishable from the other senators with nontrivial probabilities of being the most liberal, Senators Sarbanes and Reed: the probabilities that these senators have voting scores more liberal than Kerry's are both .51, or roughly 50–50. That is, while these senators each have a slightly lower probability of being the most liberal senator than does Kerry, Kerry's voting score is indistinguishable from their voting scores (for the 25 roll calls in which Kerry casts a vote—out of the 62 key roll calls analyzed by the National Journal—Kerry's voting record is indistinguishable from Sarbanes' and Reed's. Any of these three senators could be the most liberal senator of 2003.
Comparing Bush and Kerry
A second problem with the analysis thus far is that it is silent on the comparison that is arguably most politically consequential—the relative positions of President Bush and Senator Kerry. This comparison is straightforward to make: we compute a voting score for President Bush treating Bush as an additional legislator in the statistical analysis, with a voting record on the National Journal key votes given by Bush's announced positions on these votes as reported by CQ Weekly. Table 1 reveals that both Kerry and Bush failed to take a position on approximately 60% of the 62 analyzed votes, and so we expect to recover an implied voting score for Bush that has a wide confidence interval.
Figure 3 presents our estimated relative rank ordering for both the 108th Senate and President Bush using the 62 key votes of 2003 identified by the National Journal. Bush's implied voting record is at least as conservative as any Republican senator. In fact, 17 Republican senators have voting records that are in complete agreement with Bush's announced positions (ignoring abstentions). The fact that Bush announced positions relatively infrequently (on 24 of 62 key votes) generates considerable uncertainty in the voting score we assign him. In turn, this generates considerable uncertainty in the rank orderings among conservative Republicans, just as Kerry's high rate of absenteeism, coupled with a relatively extreme voting record, generates uncertainty in any assessment of the degree of Kerry's liberalism. The probability that Bush lies to the right of the entire Senate is .41; other contenders for the “most conservative senator” include Senator Mitch McConnell of Kentucky (probability of .18), Senator George Allen of Virginia (.16), and Senator Conrad Burns of Montana (.11).
Rank Ordering and 95% Confidence Intervals including President Bust using 62 National Journal Key Votes Notes
It is worth repeating that the combination of short and extremely one-sided voting histories means that estimates of legislators' voting scores (and rank orderings) are imprecise. Although the most probable rank ordering supported by the data locates Bush as more conservative than the entire Senate, a conventional 95% confidence interval on Bush's voting score extends far into Republican ranks: in fact, there is better than a 5% chance that Bush is less conservative than as many as 22 senators. Likewise, although the most probable rank ordering supported by the data locates Kerry as more liberal than the rest of the Senate, there is a better than 5% chance that Kerry is less liberal than as many as 16 senators. These confidence intervals are large when one considers that the Senate consists of 100 legislators, underscoring the limits of the available data.
A Longer Horizon: Bush and Kerry in the 107th Congress
The analysis above highlights the difficulties of estimating voting scores for incumbent legislators who are actively seeking the presidency. Campaigning takes these legislators away from Washington, generating considerably high rates of abstention in their voting records in the run up to an election. Indeed, the problem is probably more pernicious: the roll calls that do draw candidates back to Washington to cast votes are not a random subset of roll calls, but are on issues where their votes might have utmost importance for procedural reasons. Party loyalty rather than a genuine ideological position might explain some of these votes. In fact, there is some evidence that supports this notion:
As John Kerry was preparing to fly to Des Moines to unveil his health care platform on May 15, his well-laid plans for furthering his presidential candidacy were suddenly complicated by his day job representing Massachusetts in the United States Senate. The Republican leadership had chosen that day for the climactic votes on President Bush's second major tax cut—sure to be among the defining issues of the 2004 campaign. So, like the other five members of Congress seeking the Democratic presidential nomination, Kerry scrambled his campaign schedule and was on hand to vote against the bill that ultimately became last year's $330 billion, 11 year tax-cut (Kady II 2004, 22).
To determine if analyzing more votes in a non-presidential year is (both statistically and politically) consequential we analyze the 498 non-unanimous roll calls of the 107th Senate. Again, we include President Bush in the analysis, adding his announced positions to the roll call data (Bush publicly announced positions on 63 of the 498 non-unanimous roll calls). In contrast to the National Journal key votes of 2003, Kerry misses just 15 of the 498 non-unanimous roll calls in the 107th Senate.3
We also break the voting history of Senator Jim Jeffords (VT) into two pieces, corresponding to his switch from a Republican to an Independent; the data strongly suggest that his voting records changed as a result of this switch.
Figure 4 presents the rank-ordering and 95% confidence intervals that result from the analysis of this larger data set. Taking this longer time horizon and analyzing more votes in a nonpresidential election year reveals a slightly different conclusion. The probability that Kerry is the most liberal senator in the 107th Senate is infinitesimally small. Our best guess is that Kerry is the 16th most liberal senator, and a 95% confidence interval on his rank ranges from the 10th to the 20th most liberal. In contrast, the probability that President Bush is the most conservative lawmaker in the analysis is 31%; other contenders include Republican Senators Jesse Helms (NC, with probability 0.24), John Kyl (AZ, 0.20), and Phil Gramm (TX, 0.17). Our best guess is that Bush is more conservative than the entire Senate, but since the president did not take a position on every vote we cannot estimate his score (and rank) with much precision. As a result, a 95% confidence interval over Bush's rank extends from the most conservative to the 27th most conservative.
Rank Ordering and 95% Confidence Intervals including President Bust using all non-unanimous roll calls from the 107th Senate. National Journal Key Votes
In summary, with this larger set of data, the conclusion that Kerry is extremely liberal simply is not supported. To be sure, Kerry is more liberal than most senators, and we would unambiguously locate him among the most liberal quintile of the Senate. But Kerry is far from being the most liberal senator in the 107th Senate; the late Paul Wellstone (D-MN) is unambiguously to the left of the rest of the 107th Senate. In fact, the only conclusion that is supported by analyses of both the 107th Senate and the 62 votes analyzed by the National Journal in the 108th Senate is that Kerry and Bush are among the most liberal and most conservative quintiles, respectively.
Conclusion
A voting score simplifies a legislative voting history into a single number, providing an easily understood and easily communicated basis for comparing legislators. But it is important that voting scorecards be understood in perspective. A reanalysis of the 62 key votes studied by the nonpartisan National Journal are actually much more ambiguous than reported. The probability that Kerry is the most liberal senator is only 30%—far from the certainty implied by newspaper headlines. Furthermore, using conventional levels of statistical significance, we cannot rule out the possibility that Kerry is actually only the 20th most liberal senator based on the 62 votes in 2003 analyzed by the National Journal.
An analysis including the public positions taken by President Bush reveals that Bush lies in the most conservative quintile of senators. Analyses that fail to include President Bush's publicly announced positions fail to provide an appropriate context for interpreting Kerry's score. A more balanced analysis, one that includes both Bush and Kerry, finds that both are extreme relative to the Senate; Kerry is more liberal and Bush more conservative than a large majority of the Senate. This conclusion is confirmed through analysis of votes cast in the 107th Senate.
An additional point on the political context of our findings is worth noting. While it is one thing to find that a Massachusetts senator has a quite liberal voting history, it is arguably of more political significance to discover that the announced positions of the Bush administration generate an implied voting history likely to be the most conservative in the Senate. In short, the Bush administration's apparent conservatism is arguably more politically striking than Kerry's apparent liberalism if we were to compare their voting records with the preferences of the median voter in their nominal constituencies (i.e., Massachusetts for Kerry, and the entire country for Bush).
Political scientists have devoted considerable effort (e.g., Clausen 1973; Kingdon 1989; Poole and Rosenthal 1997; Clinton, Jackman, and Rivers 2004) to analyzing roll call votes and the properties of vote scores for legislators (e.g., Snyder 1992). Given a political environment in which candidates are eager to use information to portray their opponents as extreme, it is important to place any vote scores in a proper context. Just as it is problematic to report poll results without uncertainty assessments, so too is it problematic to report vote scores without accounting for the scores' uncertainty. The characterization of Sen. John Kerry as the most liberal senator resulting from the publication of the National Journal scores is an oversimplification. In fact, the evidence is a bit more ambiguous than reported, and the context, especially relative to President Bush's corresponding conservatism, is missing. One should not be surprised when journalists do not report or ask for confidence intervals. Confidence intervals are usually considered technical jargon and too complex to explain to the general public. Unfortunately, in this case, they were very relevant to the story.
