International relations scholars and foreign policy analysts typically claim to be interested in prescriptive theorizing, but this usually takes the form of a perfunctory concluding chapter about how policy makers should behave if their theoretical premises are right, which of course they are. The normative implications of academic work are generally a tacked-on epilogue. We are usually much more interested in explanation than prescription. Not so with Jeffrey Friedman’s excellent book War and Chance: Assessing Uncertainty in International Politics, the best book on improving decision making through rigorous empirical analysis since Philip Tetlock’s landmark Expert Political Judgment: How Good Is It? How Can We Know? (2005).
Friedman is particularly interested in the role that probability assessment plays in foreign policy judgment. In an extensive analysis of the explicit guidelines for assessing uncertainty provided by the US military, intelligence services, and foreign policy agencies, he finds an aversion to offering fine-grained probabilistic judgments. A qualitative and quantitative review of decision making during the first decade of US involvement in Vietnam illuminates a number of common pathologies. Elites rely on relative rather than absolute probability, asking whether certain options offer better chances of success than others, rather than estimating an absolute number. Or they engage in what Friedman calls “conditioning,” making the case for instance that success requires a particular action but not actually estimating the likelihood of success given that action. Both have the common weakness of asking whether success is actually worth it at all, given the costs. Elites might think they are carefully estimating probability, but actually they are avoiding estimating it. Collecting and analyzing all references to probability in US documents concerning the Vietnam War, Friedman reveals something dramatic: more than 90% of assessments of strategic options make this mistake. Political elites consistently avoid precise judgments about the overall likelihood of success, with (literally) fatal consequences. The policy most likely to yield success is that which usually has the most costs, and those should be taken into account, but often they are not.
In this way, Friedman’s book fits squarely in the tradition of psychological research on foreign policy making, which is united by a pessimistic assessment of the quality of judgment exercised by our leaders. However, Friedman does not leave it there, as so many scholars do. He rejects the arguments of those he calls agnostics, who claim that probability judgments are too subjective to be of any use, and rejectionists, who go so far as to argue that these judgments are counterproductive and harmful to foreign policy interests. He makes two compelling points. First, in a policy-making domain where uncertainty is always present, elites must nevertheless make decisions. They might not like uncertainty, but they cannot avoid it. So much psychologically inspired research leaves us hanging. People are terrible decision makers, yes. But knowing that, what are we supposed to do? Most research is content to simply score the theoretical points against mainstream theories by showing that human beings demonstrate systematic departures from rationality. In full disclosure, I would even count myself in this group.
Second, Friedman argues that this skepticism might be misplaced, because rarely do scholars actually test whether probabilistic judgment fails more in comparison to nonprobabilistic judgment and whether the former can be improved. This second point receives most of the attention in the book. Friedman is much more optimistic about the prospects for probabilistic judgment than the conventional wisdom, if elites would just give it a try. The book is equal parts criticism and pep talk. As he writes, “The worse our performance in this area becomes, the more priority we should place on preserving and exploiting whatever insight we usually possess” (p. 10).
Friedman first takes on the agnostics who argue that probability judgments are essentially useless because they can vary so wildly, drawing on research he undertakes using surveys of national security experts at the Naval War College, as well as forecasters involved in the Good Judgment Project organized by Tetlock and others. The data arising out of the forecasting tournament organized by the latter are the most compelling. By asking thousands of forecasters to make predictions about foreign policy events, it becomes possible to assess their quality of judgment. By asking them to also specify probabilities, it becomes possible to estimate the gain in judgment accuracy when compared to coarser estimates of the kind typically made in the foreign policy establishment, such as whether an outcome is very likely or highly improbable; that is, what the agnostics think is the most that we can expect from foreign policy makers. Friedman does this by rounding up or down probabilities offered by forecasters into broader “bins” corresponding to the number of categories typically found in guidelines for uncertainty judgment seen in the foreign policy establishment and then testing the overall “Brier” scores—an established measure of good judgment—for the coarser and more precise judgments over the course of all predictions. In other words, Friedman estimates how much they gain by being more precise, with the conventional wisdom being not very much.
The results show that it always pay to try to be more precise, with substantial increases in judgmental accuracy. Foreign policy makers may not want to make these probabilistic judgments, but they do better when they do. What agnostics might dismiss as gut decisions based on the arbitrary assignment of numbers to some underlying intuition actually appear to be much more than that. Moreover, a finer-grained analysis shows that variation in forecasting prowess is not a function of those factors that cannot be refined by foreign policy agencies, such as numerical fluency or educational attainment. Better forecasting is not a dispositional trait that is unamenable to training, effort and experience.
Of course, foreign policy judgment does not occur in a laboratory or online tournament but in a political environment where getting things wrong can come with intense political punishment. This creates incentives for those actually making these decisions for a living to avoid this very precision, using vague terms to protect themselves if their estimates turn out to be wrong. This creates the possibility for what Friedman calls elastic redefinition. Or foreign policy makers can overstate the possibilities of worst-case scenarios in what he labels strategic caution.
Friedman does not deny that foreign policy makers are punished for foreign policy failures. However, in a review of all of the major intelligence failures of the post–World War II era, he finds that using strategic caution or elastic redefinition did not preempt criticism. Noting that there has actually never been an empirical test of whether the public chastises intelligence officers more for specifying numeric probabilities as opposed to offering vaguer qualitative assessments, he undertakes a survey experiment and finds that actually publics are more forgiving when foreign policy analysts make more precise estimates and they turn out to be wrong.
In making the case for more rational decision making, Friedman provides a useful counterweight to the often overly pessimistic psychological literature, which has become the conventional wisdom in the scholarship on judgment. Given that there are no greater costs to making precise estimates and there are significant benefits, Friedman’s book ends with an admonition to rely more on explicit probability judgments. In essence, we are more capable than we give ourselves credit for, if we would give chance a chance. This is the best type of prescription, that which is based on exemplary research.
