Representing the Ideal Experiment: Unobservable Costs

An ideal experiment would manipulate protest behavior by changing the costs in a way that is unobservable to the government. So imagine a situation in which c cannot be observed by the government but can be observed by the citizen group and a researcher.

Using Bayes’ rule, if a protest occurs, the government has posterior beliefs:

$$\Pr \left( {\bar{\theta }\,|\,{\rm{protest}}} \right) = {{2p} \over {1 + p}}.$$

If a protest does not occur, the government is certain the group is of low type:

$$\Pr \left( {\bar{\theta }\,|\,{\rm{no}}\;{\rm{protest}}} \right) = 0.$$

What is the average effect on the government’s response of a decrease in the cost of protesting?

If costs are low, then a protest occurs for sure and the government’s response is $r\left( {1,{{2p} \over {1 + p}}} \right)$. If costs are high, then a protest occurs with probability p. If a protest occurs, the government’s response is again $r\left( {1,{{2p} \over {1 + p}}} \right)$. But if a protest does not occur, the government’s response is r(0, 0). So the average effect of lower costs is

(1)$$\eqalignb{ r\left( {1,{{2p} \over {1 + p}}} \right) - \left[ {pr\left( {1,{{2p} \over {1 + p}}} \right) + \left( {1 - p} \right)r\,\left( {0,0} \right)} \right] \cr = \left( {1 - p} \right)\left[ {\bar{\alpha } - \underline \alpha + \bar{\beta } \cdot {{\left( {{{2p} \over {1 + p}}} \right)}^\gamma }} \right].$$

Equation (1) represents the reduced form causal relationship between the costs of protesting and the government’s behavior (think of the analogue to two-stage least squares). Of course, lowering the cost does not always result in a change in protest behavior. With low costs, there is always protest, whereas with high costs, there is protest with probability p. Consequently, the effect of lowering of costs on the probability of protest is 1 − p. Dividing the reduced-form relationship in Equation (1) by this “first stage” gives the local total average effect of increased protest on government response that results from unobservably lowering costs (this is the analogue of the IV or Wald estimator). We label this τ _u:

(2)$${\tau _u} = \bar{\alpha } - \underline \alpha + \bar{\beta } \cdot {\left( {{{2p} \over {1 + p}}} \right)^\gamma }.$$

We can decompose this total effect into two substantive components: a direct effect and an informational effect. The direct effect is the effect of increased protest behavior holding beliefs fixed (here we hold them fixed at ${{2p} \over {1 + p}}$):

(3)$${\delta _u} = \bar{\alpha } - \underline \alpha + \left( {\bar{\beta } - \underline \beta } \right) \cdot {\left( {{{2p} \over {1 + p}}} \right)^\gamma }.$$

The informational effect is the effect of changing the government’s beliefs from what they would be if no protest occurs (0) to what they would be if a protest occurs $\left( {{{2p} \over {1 \,+\, p}}} \right)$, holding protest behavior fixed (here, fixed at a protest not occurring):

(4)$${\iota _u} = \underline \beta {\left( {{{2p} \over {1 + p}}} \right)^\gamma }.$$

The total effect (τ _u) is equal to the sum of the direct effect (δ _u) and informational effect (ι _u). Notice, we could have defined the direct effect holding beliefs fixed instead at 0 and then redefined the information effect at a protest occurring. Nothing hinges on this choice.

Importantly, τ _u, δ _u, and ι _u do not represent quantities that we think of as directly observable by an empirical researcher. Rather, these are representations of effects of theoretical interest in the ideal experiment, which an empirical researcher might never encounter. The commensurability problem is about whether actual research designs estimate any of these quantities.

Representing an Actual Research Design: Observable Costs

A common research design uses weather as an instrument, which of course, is observable to the government. So our next step is to study, within our model, the effect of an observable shock to the cost of protesting. Then, we ask whether there is a commensurability problem: that is, does the quantity that the actual research design estimates correspond to a theoretical quantity of interest in the ideal experiment?

So let’s think about the same model, but now assume that the costs are observable to the government. Using Bayes’ rule again, we now have to calculate the government’s beliefs conditional on both whether or not a protest occurs and whether costs are high or low. If costs are low, both types protest, so the government learns nothing and its posterior equals its prior:

$$\Pr \left( {\bar{\theta }\,|\,{\rm{protest}},\underline c } \right) = p.$$

By contrast, if costs are high, one type protests and the other doesn’t, so the government learns everything:

$$\Pr \left( {\bar{\theta }\,|\,{\rm{protest}},\bar{c}} \right) = 1.$$

$$\Pr \left( {\bar{\theta }\,|\,{\rm{no}}\;{\rm{protest}},\bar{c}} \right) = 0.$$

What is the average effect on the government’s response of the increased probability of protest associated with a decrease in the cost of protesting when costs are observable?

If costs are low, then a protest occurs for sure and the government’s response is r(1, p). If costs are high, then a protest occurs with probability p. If a protest occurs, the government’s response is r(1, 1), but if a protest does not occur, the government’s response is r(0, 0). So the average effect of lowering observable costs is

$$\eqalign{ r\left( {1,p} \right) - \left[ {pr\left( {1,1} \right) + \left( {1 - p} \right)r\,\left( {0,0} \right)} \right] \cr\quad = \left( {1 - p} \right)\left( {\bar{\alpha } - \underline \alpha } \right) + \bar{\beta }\left( {{p^\gamma } - p} \right).$$

Just as before, we divide by 1 − p to get the effect of the change in protest behavior due to a change in costs (τ _o):

$${\tau _{\rm{o}}} = \bar{\alpha } - \underline \alpha + \bar{\beta } \cdot {{{p^\gamma } - p} \over {1 - p}}.$$

If an empirical researcher uses a research design that corresponds to this nonideal experiment (i.e., one with observable cost shocks), then τ _o represents the resulting IV estimate.

Commensurability

It is now straightforward to think about commensurability in the context of our example. We imagine that an empirical researcher uses a research design with observable costs. Thus, the only thing she actually observes is an estimate of τ _o. But what she really wants to learn about are effects of theoretical interest in the ideal experiment—τ _u, δ _u, or ι _u. Can she use her estimate of τ _o to do so?

A first observation is that the total effect of the change in protest induced by a shock to costs is not the same when that shock is and is not observable. Formally, τ _u = τ _o if and only if

$${\left( {{{2p} \over {1 + p}}} \right)^\gamma } = \,{{{p^\gamma } - p} \over {1 - p}},$$

which never holds. The left-hand side is greater than the right-hand side for any p ∈ (0, 1) and γ > 0. This means that the research design using observable weather shocks does not estimate the total effect in the ideal experiment, a manifestation of the commensurability problem in our example.

The key thing going on is a difference in the information effects between the actual research design and the ideal experiment. In both cases, there is the same change in protest behavior in response to a cost shock—the probability of protest increases by 1 − p. But there are different effects on beliefs. When costs are observable, the informational content of a protest depends on what the costs are—the government understands that a large protest means something different on a sunny day and on a rainy day. When costs are low, there is no informational content—holding a protest on a nice sunny day is too easy for the government to conclude anything about citizen attitudes. But when costs are high, there is considerable informational content because only truly angry citizens protest in the rain. By contrast, with unobservable shocks, the government has to average across these possibilities, leading to different beliefs in the two scenarios. As a consequence, the same change in average protest behavior has different effects on government behavior in the ideal experiment and the actual research design. This creates the commensurability problem: the quantity estimated by the research design does not correspond to the theoretical object of interest in the ideal experiment. This fact corresponds to our first theorem below.

One way of thinking about this result is that an observable shock violates the exclusion restriction because it affects the way the treatment (here, protest) is interpreted. In this sense, our argument can be understood as illustrating how theoretical arguments can highlight an important way in which, to use Heckman’s (Reference Heckman2000) terminology, an external instrument may fail to be exogenous. (Sarsons [Reference Sarsons2015] discusses other ways rainfall may violate the exclusion restriction.)

A thought one might have is that, although the estimate generated by a research design with observable shocks does not correspond to the total effect of increased protest in the ideal experiment, maybe it corresponds to just the direct effect. The intuition is that when the government observes the cost shock, it knows those shocks are causing a change in protest behavior. Hence, it won’t mistakenly think that increased protest size due to lower costs is the result of a genuine change in antigovernment sentiment. So, maybe once it takes the shock into account, there is no informational content left in the change in protest behavior. If this is the case, perhaps the effect being estimated by a research design using observable shocks to protest costs captures just the direct effect in the ideal experiment, purged of any informational effect.

Comparing the total effect in the model with observable shocks to just the direct effect in the model with unobservable shocks, and rearranging, we see that they are equal if and only if

$${{{{\left( {1 + p} \right)}^\gamma }\left( {{p^\gamma } - p} \right)} \over {{{\left( {2p} \right)}^\gamma }\left( {1 - p} \right)}} = {{\bar{\beta } - \underline \beta } \over {\bar{\beta }}}.$$

The left-hand side of this condition is strictly decreasing in γ. The right-hand side is constant in γ. As such, for any p ∈ (0, 1), this condition holds for at most one value of γ, which is to say that the actual research design isolates the direct effect in the ideal experiment only in a knife-edge case.

To start to get some intuition for what is going on, think about the case where the direct and informational effects of protest on government behavior are additively separable (i.e., $\bar{\beta } = \underline \beta$). In this case, the right-hand side is zero and the unique γ for which the condition holds is γ = 1. That is, if the direct and informational effects of protest on government behavior are additively separable, then the total effect in the actual research design equals the direct effect in the ideal experiment if and only if the government’s behavior is linear in its beliefs. Why is this true?

Additive separability implies that the direct effect doesn’t depend on the beliefs. This means that the direct effect in the actual research design and the ideal experiment are the same, even though the government always has different beliefs in those two settings. As such, for the total effect in the actual research design to be equal to the direct effect in the ideal experiment, there must be no information effect in the actual research design. It might seem that this is impossible because the government does learn from observing protest size. But, as must be the case for a Bayesian actor, the government’s posterior beliefs on average equal its prior beliefs. And so the information effect, which averages across the possible shocks, is zero if and only if beliefs enter linearly, so that the average is all that matters.

If there is not additive separability (i.e., $\bar{\beta } \ne \underline \beta$), then there is in fact heterogeneity in the direct effect itself because it depends on the government’s beliefs.^{Footnote 5} So now the direct effects in the actual research design and the ideal experiment are not equal. As such, for the total effect in the actual research design to be equal to the direct effect in the ideal experiment, it must be that the information effect in the actual research design is of precisely the right size and sign to exactly off-set the difference between the direct effects in the actual research design and in the ideal experiment. This is obviously a knife-edge condition, reflected in the fact that it holds for one and only one value of γ.

Our second theorem establishes that this logic generalizes. The quantity estimated by some research design in which the government’s information differs in any way from the ideal experiment only corresponds to the direct effect in the ideal experiment if: (1) there is no informational effect in the actual experiment and (2) the government’s best response is additively separable in the direct effect of protest and it’s beliefs about the state of the world.

Contrasts and Effects

The shock plays a critical role in our analysis, specifically, its role is to exogenously vary the outcome of the social process, A, thus changing treatment independently of the state of the world θ. We start by using the shock to define causal effects, but before doing so, it will be useful to introduce a little terminology.

Definition 1. A contrast is a pair (ω′, ω″) where ω′, ω″ ∈ Ω such that ω′ ≠ ω″. The set of contrasts is

$${\cal C} = \left\{ {\left( {\omega \prime ,\omega ''} \right) \in \Omega \; \times \;\Omega \,|\,\omega \prime \ne \omega ''} \right\},$$

endowed with the product measure.

We assume that shocks matter for treatment on average. That is, there must be a “first stage relationship” or there is no effect to study. So, for all (ω′, ω″),

$${{\mathbb{E}}_{\theta ,\varepsilon }}\left[ {T_{\theta ,\varepsilon }^{\omega \prime \prime } - T_{\theta ,\varepsilon }^{\omega \prime }} \right] \ne 0.$$

A contrast comprises two distinct values of the shock. The local average total effect of a change in treatment due to a contrast is obtained by comparing the decision-maker’s average response at two distinct values of the shock divided by the average value of treatment at the same two values of the shock:

Definition 2. The local average total effect at the contrast $\left( {\omega \prime ,\omega ''} \right) \in {\cal C}$ is

$$\eqalign{ \tau \left( {\omega \prime ,\omega ''} \right) \cr= {{{{\mathbb{E}}_{\theta ,\varepsilon }}\left[ {r\left( {T_{\theta ,\varepsilon }^{\omega ''},\;\pi \left( {I\left( {\theta ,\omega '',\varepsilon } \right)} \right)} \right) - r\left( {T_{\theta ,\varepsilon }^{\omega '},\;\pi \left( {I\left( {\theta ,\omega ',\varepsilon } \right)} \right)} \right)} \right]} \over {{{\mathbb{E}}_{\theta ,\varepsilon }}\left[ {T_{\theta ,\varepsilon }^{\omega ''} - T_{\theta ,\varepsilon }^{\omega '}} \right]}}.$$

The total effect can be decomposed into a direct effect and an informational effect.

Definition 3. Fix a posterior belief $\pi _{\theta ,\varepsilon }^{\omega '} = \pi \left( {I\left( {\theta ,\omega ',\varepsilon } \right)} \right)$. The local average direct effect at the contrast (ω′, ω″) and the belief $\pi _{\theta ,\varepsilon }^{\omega '}$ is

$$\delta \left( {\omega \prime ,\omega ''} \right) = {{{{\mathbb{E}}_{\theta ,\varepsilon }}\left[ {r\left( {T_{\theta ,\varepsilon }^{\omega ''},\pi _{\theta ,\varepsilon }^{\omega '}} \right) - r\left( {T_{\theta ,\varepsilon }^{\omega '},\pi _{\theta ,\varepsilon }^{\omega '}} \right)} \right]} \over {{{\mathbb{E}}_{\theta ,\varepsilon }}\left[ {T_{\theta ,\varepsilon }^{\omega ''} - T_{\theta ,\varepsilon }^{\omega '}} \right]}}.$$

Definition 4. Fix a treatment status $T_{\theta ,\varepsilon }^{\omega ''}$. The local average information effect at the contrast (ω′, ω″) and the treatment status $T_{\theta ,\varepsilon }^{\omega ''}$ is

$$\eqalign{\iota \left( {\omega ',\omega ''} \right) \cr= {{{{\mathbb{E}}_{\theta ,\varepsilon }}\left[ {r\left( {T_{\theta ,\varepsilon }^{\omega ''},\pi \left( {I\left( {\theta ,\omega '',\varepsilon } \right)} \right)} \right) - r\left( {T_{\theta ,\varepsilon }^{\omega ''},\pi \left( {I\left( {\theta ,\omega ',\varepsilon } \right)} \right)} \right)} \right]} \over {{{\mathbb{E}}_{\theta ,\varepsilon }}\left[ {T_{\theta ,\varepsilon }^{\omega ''} - T_{\theta ,\varepsilon }^{\omega '}} \right]}}.$$

From Definitions 2–4, we have the following decomposition:

(5)$$\tau \left( {\omega ',\omega ''} \right) = \delta \left( {\omega ',\omega ''} \right) + \iota \left( {\omega ',\omega ''} \right)$$

(6)$$= - \left( {\delta \left( {\omega '',\omega '} \right) + \iota \left( {\omega '',\omega '} \right)} \right).$$

Notice that for a single contrast, (ω′, ω″), there are two (potentially) different direct and informational effect pairs whose sum corresponds to the same total effect. Nothing hinges on which pair one chooses.

Experiments

Our definition of an experiment is in the spirit of Blackwell (Reference Blackwell1951). Our framework describes a set of states corresponding to the cross-product of the sets of contrasts, states of the world, and idiosyncratic noise. In our terminology, an experiment, ${\cal E}$, is the probability measure over this set of states, along with the decision-maker’s information set (which we denote ${I^{\cal E}}$). When we compare experiments (representing various research designs), we hold the probability measure over the set of states fixed and vary the decision-maker’s information set.

To start thinking about commensurability, we first need to define the ideal experiment. As we discussed in the introduction, and illustrated in our example, an ideal experiment is one where the decision-maker observes treatment but does not observe the process that leads to her particular treatment assignment.

Definition 5. The ideal experiment is an experiment where the decision-maker’s information set is ${I^{\cal I}}\left( {\theta ,\omega ,\varepsilon } \right) = \left\{ {T_{\theta ,\varepsilon }^\omega } \right\}.$

Commensurability

We are interested in whether various nonideal experiments, corresponding to canonical research designs used in prominent empirical literatures, might estimate an effect of theoretical interest in the ideal experiment. To answer this question, we first need to develop some terminology.

Label the total, direct, and information effects at a contrast (ω′, ω″) in some experiment ${\cal E}$ by $\tau _{\theta ,\varepsilon }^{\cal E}\left( {\omega ',\omega ''} \right)$, $\delta _{\theta ,\varepsilon }^{\cal E}\left( {\omega ',\omega ''} \right)$, and $\iota _{\theta ,\varepsilon }^{\cal E}\left( {\omega ',\omega ''} \right)$, respectively. For the ideal experiment in particular, we label these quantities $\tau _{\theta ,\varepsilon }^{\cal I}\left( {\omega ',\omega ''} \right)$, $\delta _{\theta ,\varepsilon }^{\cal I}\left( {\omega ',\omega ''} \right)$, and $\iota _{\theta ,\varepsilon }^{\cal I}\left( {\omega ',\omega ''} \right)$.

Conceptually, one can think of any given study as being represented by one realization of some experiment in our framework. So we start by defining what it means for one particular instance of an experiment, associated with some particular contrast, to capture an effect of interest in the ideal experiment. The idea is that the experiment captures the effect in the ideal experiment if the total effect in the experiment (which represents the only thing the empirical researcher observes) equals the effect of interest in the ideal experiment.

Definition 6. An experiment, ${\cal E}$, captures an effect ${E^{\cal I}} \in \left\{ {{\tau ^{\cal I}},{\delta ^{\cal I}},{\iota ^{\cal I}}} \right\}$ in the ideal experiment at the contrast (ω′, ω″) and prior p if

$${\tau ^{\cal E}}\left( {\omega ',\omega ''} \right) = {E^{\cal I}}\left( {\omega ',\omega ''} \right).$$

Of course, an experiment capturing an effect of theoretical interest in the ideal experiment at one particular contrast and prior is not enough to have confidence that we are learning about the quantity of interest. For that, we want a theoretical analogue of econometric identification (Imbens and Angrist Reference Imbens and Angrist1994; Koopmans and Reiersol Reference Koopmans and Reiersol1950). That is, we want to ensure that an actual research design captures the effect of interest in the ideal experiment with probability one.

Definition 7. An experiment, ${\cal E}$, is commensurate with the ideal experiment for an effect ${E^{\cal I}} \in \left\{ {{\tau ^{\cal I}},{\delta ^{\cal I}},{\iota ^{\cal I}}} \right\}$ if ${\cal E}$ captures the effect ${E^{\cal I}}$in the ideal experiment at almost every contrast and prior $\left\langle {\left( {\omega ',\omega ''} \right),p} \right\rangle \in {\cal C} \times \left( {0,1} \right)$.

Discussion of Concepts

In this section, we discuss two important components of our analysis. First, we discuss our choice to study the commensurability problem in the context of an ideal experiment. Second, we briefly relate our assumptions to common theoretical frameworks.

Haavelmo (Reference Haavelmo1944, 14) distinguishes between two different classes of experiments, an ideal hypothetical experiment that isolates causal channels and the set of experiments that are naturally produced by Nature. Our analysis is about the relationship between these two types of experiments. Building on this classic distinction, the benchmark of the ideal experiment is endorsed by Angrist and Pischke (Reference Angrist and Pischke2009, 1) as a way of precisely articulating causal questions, with a particular emphasis on assessing the impact of interventions.

An important challenge to causal studies is articulated by Deaton (Reference Deaton2010) and Heckman (Reference Heckman2000, 84–6), who stress the importance of using a structural model as a benchmark to evaluate the local average treatment effect because, they argue, structural parameters have a clear substantive interpretation. They further argue that because most studies do not link the local average treatment effect to a structural model, a theoretical interpretation of the LATE is not implied.

We, of course, agree that learning about structural parameters is the most satisfying version of estimating quantities of theoretical interest. Our approach, however, allows for the possibility that there are reduced-form quantities, short of structural parameters, that might nonetheless qualify as of theoretical interest. For instance, in a study of protests, one might want to answer a question like, “if the leader of an antigovernment group were to switch to using violent tactics, would the movement become more or less effective?” Obviously, if one knew the structural parameters of the right model of protest, one could answer this question. But this is more demanding than is needed. One could also answer this question with the total effect from an ideal experiment. Hence, we define the ideal experiment and the associated effects as we do to give the reduced-form approach the best shot possible at identifying quantities of theoretical interest even when it cannot identify structural parameters.

To highlight our results, we develop our framework abstractly, rather than providing a more detailed behavioral or game-theoretic structure. We do this for a number of reasons. First, our results follow from general properties, namely, that the decision-maker responds to treatment status and beliefs.^{Footnote 7} Second, our results are consistent with a number of different substantive scenarios. Generality, emphasize that many different game forms are consistent with our approach. Finally, deriving these features of the environment as part of a game form would involve a number of additions to our setup, such as payoffs and solution concepts, all of which are standard and would not influence our main results, but would complicate exposition.

The Efficacy of Protest

Protest and other forms of mobilization can influence government policy—be it repression, concessions, or other responses—through a multitude of different channels, many of which fit into the distinction between the direct and informational channels highlighted above. For instance, mobilized dissent, as an expression of political preferences, can communicate information to the government about the extent of political dissatisfaction in society, and, thus, may have an informational effect on the government’s response. In addition, protests may be disruptive to the functioning of society or the economy, and consequently, governments may respond to such dissent even in the absence of new information about citizen dissatisfaction, thus giving rise to a direct effect.

Assessing the efficacy of various forms of political mobilization is a central, and vexing, concern for empirical studies. To assess the efficacy of political mobilization, one needs to compare government responses with various levels or types of mobilization. The fundamental problem, of course, is that the decision to mobilize is endogenous.

Not surprisingly, empirical work has turned to searching for sources of exogenous variation in political mobilization. The most prevalent strategy is an instrumental variables approach (Collins and Margo Reference Collins and Margo2007; Dell Reference Dell2012; Henderson and Brooks Reference Henderson and Brooks2016; Hendrix and Salehyan Reference Hendrix and Salehyan2012; Huet-Vaughn Reference Huet-Vaughn2013; Madestam et al. Reference Madestam, Shoag, Veuger and Yanagizawa-Drott2013; Ritter and Conrad Reference Ritter and Conrad2016). The basic idea is to look for factors that shift individuals’ propensity to mobilize, but that have no independent channel to affect outcomes. The most common source of variation used in this literature comes from the weather. The idea is that if a protest is scheduled for a certain date, and on that date there is, say, unexpected torrential rain or unusually high temperatures, the individual cost of participating in the protest increases. Thus, these random natural phenomena provide an exogenous shock to citizens’ costs of participating in a protest, which can be used in an instrumental variables analysis to isolate the effect of mobilized turnout on government behavior.

Representing this approach in our framework is straightforward. The government is the decision-maker, and the government’s response is measured, for example, by levels of repression. The state of the world, θ, corresponds to the amount of antigovernment sentiment among citizens, and the social process, A, corresponds to the aggregate level of participation in antigovernment activities by citizens. In this case, we can think of the treatment as simply being the level of protest, so that T(A) = A.

The amount of protest is determined by the level of antigovernment sentiment (θ), the weather on the day of the protest (ω), and other idiosyncratic factors (ε). In this context, variation in weather corresponds to a contrast, (ω′, ω″). The effect of interest is isolated by measuring the association between variation in the level of protest due to rainfall and variation in repression (or, in the case of the reduced-form relationship, the association between variation in rainfall and variation in repression). Many standard models in the literature on protest show how to micro-found protest in these kinds of concerns (Angeletos and Pavan Reference Angeletos and Pavan2013; Angeletos, Hellwig, and Pavan Reference Angeletos, Hellwig and Pavan2006, Reference Angeletos, Hellwig and Pavan2007; Bueno de Mesquita Reference Bueno de Mesquita2010; Casper and Tyson Reference Casper and Tyson2014; Edmond Reference Edmond2013; Shadmehr and Bernhardt Reference Shadmehr and Bernhardt2011; Tyson and Smith Reference Tyson and Smith2018).

Empowering Female Candidates

Our second application corresponds to empirical studies that focus on the empowering effect of women officeholders by addressing whether one female candidate winning elected office causes other potential female candidates to run for office in subsequent elections (Baskaran and Hessami Reference Baskaran and Hessami2018; Broockman Reference Broockman2014; Ladam, Harden, and Windett Reference Ladam, Harden and Windett2018).

A victory by a female candidate could empower other female candidates through both direct and informational channels. A direct effect might arise if women prefer serving in office when there are other elected women who can, for example, serve as mentors or who may be desirable legislative collaborators. An informational effect might derive from potential female candidates interpreting earlier electoral victories by female candidates as evidence that voters are less discriminatory than they had previously suspected.

Identifying these causal effects is, of course, difficult because of a variety of endogeneity concerns—for instance, perhaps female candidates both win and emerge more often in more liberal districts. To address such concerns, a growing literature uses an election regression discontinuity design (Baskaran and Hessami Reference Baskaran and Hessami2018; Broockman Reference Broockman2014). The idea is to consider districts with male and female candidates running against one another. In some, the female candidate just barely wins, and in others, the male candidate just barely wins. If potential outcomes (i.e., future female candidates’ willingness to run) are continuous at the electoral threshold, then comparing the frequency with which future female candidates run in districts that had a female vs. male victory provides an estimate of the causal effect of having a female candidate win on future female candidacies.

Again, we can represent this scenario within our framework. The decision-maker in our model corresponds to a future potential female candidate. We can think of her response function, r, as measuring her willingness to run. The state of the world in the model, θ, is the amount of antifemale bias among the electorate, and the outcome of the social process, A, corresponds to the vote differential between the female and male candidate in the initial election. It is determined by antifemale bias (θ), other features of the candidates or electoral environment that affect voting for the two candidates (ω), and idiosyncratic factors on election day (ε).

The treatment is the gender of the electoral winner in the initial election. This treatment is a monotone function of the vote differential, A, but it changes values only once, at the electoral threshold. In particular, treatment can take one of two values—a female (F) or male (M) winner in the previous election:

$$T\left( A \right) = \left\{ \matrix{{F \ {\rm if}\;A \,\ge \,0} \cr M \ {{\rm{if}}\;A < 0.} \cr } } \right.$$

The effect of interest is isolated by comparing districts where a female candidate just won against a male opponent to districts where a male candidate just won against a female opponent.

Indiscriminate Violence in Counterinsurgency

The final application we focus on concerns the effects of indiscriminate violence by counterinsurgents on rebel violence. Naturally, there are important empirical challenges that arise in isolating the effect of indiscriminate violence. The most common research design in the indiscriminate violence literature uses a difference-in-differences approach to compare matched localities in periods when they did or did not experience indiscriminate violence (Benmelech, Berrebi, and Klor Reference Benmelech, Berrebi and Klor2014; Condra and Shapiro Reference Condra and Shapiro2012; Lyall Reference Lyall2009). For instance, Lyall (Reference Lyall2009) exploits a Russian doctrine of random shelling to generate variation in civilian casualities among similar villages and Condra and Shapiro (Reference Condra and Shapiro2012) match on levels of violence and exploit civilian deaths resulting from the violence.

Dell and Querubín (Reference Dell and Querubín2017) take a somewhat different approach. They study the effect of aerial bombing by American forces in Vietnam on violent activity by the North Vietnamese insurgents (among other outcomes). The source of variation in their study comes from discontinuities in U.S. bombing strategy. Each locality was given a score based on a complex algorithm whose inputs included regular surveys of local commanders. Rounding thresholds in that algorithm led otherwise similar hamlets to experience different levels of bombing. We focus on Dell and Querubín’s (Reference Dell and Querubín2017) study, but return to compare their results with other studies later.

It is again straightforward to map this setting to our framework. The decision-maker in our model corresponds to North Vietnamese insurgent commanders, and the response function, r, measures the amount of violence engaged in by insurgents. The state of the world in the model, θ, might represent something like the level of U.S. resolve in the Vietnam War. The social process, A, corresponds to the level of bombing used by U.S. forces against a particular hamlet. It is determined by something like the level of U.S. commitment in Vietnam (θ), the dozens of factors that affect the hamlet’s score in the bombing algorithm (ω), and idiosyncratic factors like data entry errors (ε).

The treatment is the amount of civilian casualties and destroyed infrastructure in the hamlet, T(A), which is increasing in the amount of bombing. The effect of interest is isolated using a regression discontinuity at the rounding threshold, which compares hamlets that had essentially identical scores in the bombing algorithm but experienced very different levels of bombing.

Efficacy of Protest

The application of Theorem 1 to the empirical literature on the efficacy of political mobilization is straightforward. The ideal experiment for understanding the effect of protest behavior on government response involves a shock to protest behavior that is not observable by the government. But the actual research designs used in the literature exploit variation in protest behavior due to weather shocks. Such studies may be estimating perfectly sensible causal estimates; after all, weather shocks do manipulate protest size. But weather shocks are also observable to the government (that is, $\omega \in {I^{\cal E}}\left( {\theta ,\omega ,\varepsilon } \right)$). This affects how the government updates its beliefs in response to changes in protest behavior. As such, the estimates of the efficacy of protest obtained using weather shocks do not estimate the total effect of a change in protest behavior in the ideal experiment. Put differently, they do not estimate the effect of a change in protest behavior that is due to, say, actual changes to the underlying level of citizen discontent or unobservable changes in the capacity of antigovernment groups.

There is something at stake here. Many of the most prominent contributions in this literature are interested in the question of whether protests are productive or counterproductive. There may, of course, be competing effects. Suppose the direct effect of protests is counterproductive (e.g., governments get angry at protestors and want to repress them) but the informational effect is positive (e.g., when governments learn their citizens are truly angry, they may be inclined to change policy in ways that are beneficial to the citizens). The estimate from a research design that does not mimic the informational environment of the ideal experiment, and thus estimates a combination of a direct effect and an information effect that is infected by the government’s interpretation of what the size of a protest means in light of the weather, and may get the sign of the total effect wrong relative to the ideal experiment. This could lead to incorrect, or at least difficult to interpret, conclusions about the efficacy of protest as a means to achieving political change by citizens.

Theorem 2 also implies that research designs using weather shocks do not estimate the direct effect of protest in the ideal experiment. To estimate the direct effect, there would need to be no information for the government from a change in protest size induced by weather. And, moreover, the way that the government responded to a disruptive protest would need to be additively separable from its assessment of its citizens’ level of antigovernment sentiment.

The question of additive separability is, of course, a substantive one. However, it seems unlikely that a government’s direct response to protestors in the streets is independent of its overall sense of public dissatisfaction.

We can provide a bit more formal guidance for whether there is zero information effect in the nonideal experiment where the decision-maker observes the shock. Let ϕ be the (differentiable) density function of idiosyncratic noise, ε. Then, assuming the decision-maker updates her beliefs using Bayes’ rule, her posterior belief that $\theta = \bar{\theta }$ at a level of protest A and a shock ω (with both observable) is

$$\eqalign{\pi \left( {\left\{ {A,\omega } \right\}} \right) \cr= {{p\phi \left( {{F^{ - 1}}\left( A \right) - \bar{\theta } - \omega } \right)} \over {p\phi \left( {{F^{ - 1}}\left( A \right) - \bar{\theta } - \omega } \right) + \left( {1 - p} \right)\phi \left( {{F^{ - 1}}\left( A \right) - \underline \theta - \omega } \right)}}.$$

For the information effect to be zero, for almost every A, this posterior belief needs to be constant in ω almost everywhere. Differentiating, a zero information effect requires that for almost all ω:

$${{\phi \prime } \over \phi }\left( {{F^{ - 1}}\left( A \right) - \bar{\theta } - \omega } \right) = {{\phi \prime } \over \phi }\left( {{F^{ - 1}}\left( A \right) - \underline \theta - \omega } \right).$$

This condition holds if and only if ϕ is almost everywhere log-linear, a highly restrictive condition.

We want to emphasize that we do not intend these results as critiques of the specific papers using weather shocks to estimate the efficacy of protests. Seen from the perspective of the project of doing better causal inference, those papers make important advances. Rather, our goal is to shed some light on whether we can interpret those estimates in terms of important theoretical quantities in the ideal experiment to highlight the conceptual difficulties associated with studying the effect of behavior on behavior.

Empowering Female Candidates

The best identified empirical studies on the empowerment of female candidates use a regression discontinuity design (Baskaran and Hessami Reference Baskaran and Hessami2018; Broockman Reference Broockman2014).

In the ideal experiment, the decision-maker (a future potential female candidate) observes only treatment status—i.e., whether a male or female won the last election. Hence, in the ideal experiment, the decision-maker has posterior beliefs:

$$\pi \left( {I^{\cal I} \left( {\theta ,\omega ,\varepsilon } \right)} \right) = \left\{ {\matrix{ {\Pr \left( {\theta = \bar \theta |A_{\theta ,\varepsilon }^\omega \ge 0} \right)} \ {{\rm{if}}\;T_{\theta ,\varepsilon }^\omega = F} \cr {\Pr \left( {\theta = \bar \theta |A_{\theta ,\varepsilon }^\omega < 0} \right)} \ {{\rm{if}}\;T_{\theta ,\varepsilon }^\omega = M.} \cr } } \right.$$

But now consider the actual electoral regression discontinuity design used to assess the effect of a female victory on future female candidacy. Presumably, potential future female candidates are paying some attention to nearby races precisely because such races are informative about their own electoral prospects. Hence, it seems likely that she observes not just who won the previous election (T) but the vote totals in that election as well (A). That is, $A_{\theta ,\varepsilon }^\omega \in {I^{\cal E}}\left( {\theta ,\omega ,\varepsilon } \right)$.

Notice, the binary variable T (whether a female candidate wins) is entirely determined by the continuous variable A (the vote differential). So treatment status is certainly not a sufficient statistic for the decision-maker’s information with respect to θ (voter bias). Indeed, instead, A is a sufficient statistic for T. That is, the decision-maker can ignore who won in forming her posterior beliefs; all she needs to know is the vote differential. Who won still matters for her action, but only because of direct effects. Thus, even before discussing any of the specifics of the regression discontinuity, we already know by Theorem 1 that the actual research design is not commensurate with the ideal experiment for the total effect.

Nonetheless, it will be instructive to work through what does happen in an experiment that represents the actual research design, so we can think more carefully about the regression discontinuity design and the direct effect in the ideal experiment. Given that she observes the vote total from the previous election, a Bayesian decision-maker has posterior beliefs:

$$\pi \left( {{I^{\cal E}}\left( {\theta ,\omega ,\varepsilon } \right)} \right) = {\rm{Pr}}\left( {\theta = \bar{\theta }\,|\,A_{\theta ,\varepsilon }^\omega } \right).$$

The regression discontinuity design, of course, estimates behavior at the electoral threshold A = 0. In this limit, the decision-maker’s posterior belief goes to

$$\Pr \left( {\theta = \bar{\theta }\,|\,A_{\theta ,\varepsilon }^\omega = 0} \right).$$

This posterior belief is never equal to the posterior belief in the ideal experiment. That is, T is not sufficient for (T, A), and hence by Theorem 1, the research design does not estimate the total effect in the ideal experiment.

As before, this is not a critique of the research design. In our model, continuity of potential outcomes at the threshold is satisfied. Rather, the problem is that future potential candidates know the electoral returns from the prior election, so they have more information than the decision-maker in the ideal experiment.

The fact that, when vote totals are observed, posterior beliefs are constant in treatment in the RD (i.e., in the limit beliefs are always $\Pr \left( {\theta = \bar{\theta }\,|\,A = 0} \right)$) raises the possibility that, despite not estimating the total effect in the ideal experiment, the election RD may isolate the direct effect. In particular, the electoral RD satisfies the first requirement of Theorem 2—the information effect in the nonideal experiment that represents the research design is zero because beliefs are constant in treatment.

Theorem 2 then says that whether the election RD identifies the direct effect of female victory in the ideal experiment comes down to a substantive question—whether female candidates’ willingness to run is an additively separable function of treatment status (whether a woman won) and posterior beliefs about antifemale bias in the electorate. This question, put another way, asks whether the direct effect of having a female candidate win on another woman’s willingness to run is different depending on that potential candidate’s beliefs about the amount of antifemale bias in the electorate. There are reasonable arguments in either direction. One possibility is that it is simply more productive or rewarding to serve in office when there are other women in office, regardless of what voters think. This would suggest that additive separability is plausible. Another possibility is that, say, when voters are particularly biased, the sense of solidarity associated with another politician who has also overcome considerable bias is heightened, suggesting that additive separability does not hold. We don’t take a stand regarding which assumption is right. Rather, we simply want to emphasize that Theorem 2 highlights this substantive question, which one must answer to know whether the election RD identifies a quantity of theoretical interest in the ideal experiment—namely, the direct effect of a female victory on future female candidates’ willingness to run.

This discussion of additive separability raises a final point about Theorem 2 regarding heterogeneous effects. There is no question that when the decision-maker observes vote shares, the regression discontinuity design yields an unbiased estimate of a local average direct effect at the electoral threshold—potential outcomes are continuous and beliefs are constant at the threshold. This is true whether or not the decision-maker’s response satisfies additive separability. Yet, without additive separability, this effect is not commensurate with the direct effect at the same electoral threshold in the ideal experiment.

This is because, when additive separability fails, there are heterogeneous direct effects across different beliefs. But the commensurability problem here is not the standard concern that local average treatment effects need not be the same as the average treatment effect when effects are heterogenous. Here, we are comparing two local direct effects (in the nonideal and ideal experiments), holding the localness constant—in both cases, we are talking about the direct effect at the same electoral threshold. The commensurability problem comes from the fact that, without additive separability, the local direct effects at that same threshold are different quantities in the nonideal and ideal experiments because the decision-maker’s beliefs are different.

These theoretical observations are, we think, particularly salient in light of recent evidence from election RD studies that finds null effects, suggesting that female victories do not empower future female candidates (Broockman Reference Broockman2014). The implication of Theorem 1 is that we should be careful about overinterpreting such a finding. The election RD purges the estimate of any informational effect of a female victory. Thus, at most, what we can conclude from such a finding is that there is no evidence of a direct effect of a female victory on empowerment of future female candidates. There still might be an informational effect that is washed out by the research design. Theorem 2 says that we should even be hesitant about that conclusion if we do not believe that the direct and informational components of a potential candidate’s decision are additively separable.

Indiscriminate Violence in Counterinsurgency

The application of Theorem 1 to Dell and Querubín’s (Reference Dell and Querubín2017) study of indiscriminate bombing in Vietnam leads to different conclusions than in the previous applications we consider. Recall, in relating our model to the empirical setting, we said that ω represents the inputs to the bombing algorithm scores, and contrasts correspond to small changes to score inputs that got magnified by rounding. Those scores were created by an incredibly complex process that took a huge amount of subjective data and returned simple indexes. The rounding built into that process was unobservable to the insurgents (and, indeed, even to the Americans). Hence, Theorem 1 implies that the effect estimated by Dell and Querubín (Reference Dell and Querubín2017) is commensurate with the total effect of bombing in the ideal experiment—treatment assignment is a sufficient statistic for the insurgents’ information relative to the state of world, which represents U.S. resolve or strategy in Vietnam. As such, the effect of variation in bombing due to score rounding should have the same effect on the insurgents’ beliefs, and thus response, as similar variation in bombing due to actual changes in America’s underlying resolve or strategy.

Something similar can be argued for the effect identified by Lyall (Reference Lyall2009) using random shelling by Russian artillery. Assuming Chechen rebels were unaware of the Russian military doctrine that called for random shelling, the effect of increased shelling in that setting is similar to the effect of increased bombing due to rounding errors in Vietnam. By contrast, the effect identified by Condra and Shapiro (Reference Condra and Shapiro2012) is quite different. In particular, their identification strategy exploits random collateral damage, holding fixed the amount of counterinsurgent activity. In that setting, if insurgents are aware of the amount of counterinsurgent activity, then it seems there is no extra informational content in whether a civilian happened to be killed. This, then, appears more similar to studies using the election RD to study female empowerment. The research design purges information effects. Hence, Condra and Shapiro’s (Reference Condra and Shapiro2012) estimates are not commensurate with the total effect, but by Theorem 2, they may be commensurate with the direct effect if the direct and informational effects of civilian casualties on insurgency are additively separable.