3.4.1 The Use of Real-world Distributions

As in the vast majority of conjoint analyses, Ono and Burden (Reference Ono and Burden2019) randomize factors independently by choosing each level with equal probability. This produces a uniform distribution in which all attribute combinations are equally likely. While the uniform distribution is commonly used in applications of the conjoint analysis, the corresponding real-world distribution of attributes are rarely uniform.

Indeed, the uniform randomization produces highly unusual profiles. For example, two-thirds of Republican candidate profiles will have abortion positions of “neutral” or “pro-choice.” The difference between this distribution and the one that of actual Republican politicians is stark. Using a legislator scorecard produced by the National Right to Life Council, a conservative nonprofit that advocates for pro-life policies, only 1 of the 296 Republican legislators (0.33%) could be classified as pro-choice and only 2 (0.67%) as neutral. A similar pattern emerges for Democrats. Two-thirds of presented candidate profiles take a value of neutral or pro-life, yet similarly low percentages of Democratic politicians hold those positions.

The case of abortion position may be especially dramatic, but the real-world distributions of nearly all of the attributes presented in Table 1 differ markedly from the uniform distribution. As a target profile distribution, we use data of actual legislators in the 115th Congress and compute the real-world joint distribution of 12 of the 13 attributes examined in Ono and Burden (Reference Ono and Burden2019). We do not produce the distribution of the Trait attribute due to its highly subjective quality and thus keep the uniform distribution for it. Because party is strongly correlated with nearly all remaining attributes, we consider the target profile distributions of Republican and Democrat politicians separately. Supplemental Appendix B includes details about the construction of this joint distribution.

Figure 2 shows that the marginal distributions of actual politicians’ characteristics (gray bars for Democrats, shaded bars for Republicans) differ substantially from the uniform distribution (white bars). In the case of the gender, which is the focus of the original analysis, neither the Republican nor Democratic distributions resemble the uniform: only 10.2% of Republicans and 32.2% of Democratic legislators are female. We find a similar pattern for the remaining attributes. The difference is most pronounced for the attributes that are likely to be salient to subjects, such as race and major policy positions. This suggests that the uAMCE may significantly differ from the pAMCE.

Figure 2 Experimental and target profile distributions of factors in Ono and Burden (Reference Ono and Burden2019). We compare the uniform distribution used in the original experiment and two real-world distributions of politicians’ characteristics and policy positions; Republican and Democrat legislators.

Finally, we note that the original experiment considers hypothetical political candidates. Thus, the ideal target profile distribution would be the real-world distribution of the attributes for all candidates, not only for elected legislators. Unfortunately, because the original conjoint experiment was not designed with fidelity to the real-world distribution in mind, there are many factors for which it is not possible to gather corresponding real-world distributions using data from all candidates. As a result, we use politicians in the 115th Congress as our main target profile distribution, for whom we were able to collect real-world distributions for most factors (as visualized in Figure 2).

In Section 5.1, we consider the robustness of the pAMCE estimates by replacing profile distributions of race, gender, and experience, based on publicly available candidate-level datasets. We also consider different theoretically relevant profile distributions on policy dimensions. Even when it is infeasible to collect the real-world distribution of all factors for all candidates, it is critical to take into account more realistic profile distributions and improve the external validity of conjoint analysis.

3.4.2 The Use of Counterfactual Distributions

Peterson (Reference Peterson2017) is primarily interested in how the effect of copartisanship changes according to the amount of other relevant information about candidates. Therefore, our analysis focuses on the first two steps of the original randomization—randomizing the number of factors to show and then randomly selecting which factor to present given the selected number of factors to be shown. Because each randomization uses the uniform distribution, every factor is equally likely to be shown. In particular, the marginal probability of each factor being shown is a little above $50\%$ (see Figure 3). If researchers use the widely used linear regression estimator (Hainmueller et al., Reference Hainmueller, Hopkins and Yamamoto2014), the resulting estimate of the AMCE represents the causal effect of copartisanship while averaging over low, medium, and high information environments.

Figure 3 Original and counterfactual distributions of factors in the information experiment (Peterson, Reference Peterson2017). We compare the distribution used in the experiment and two counterfactual distributions of information environment.

Rather than averaging over different information environments that have distinct substantive meanings, we may be interested in investigating how the pAMCE depends on different information environments. In particular, we consider two counterfactual distributions: a low information environment in which subjects observe each factor (other than copartisanship) only $20$ % of the time, and a high information environment in which each factor is observed $80$ % of the time. Figure 3 compares these low- and high-information counterfactual distributions to the one used in the original analysis. As the figure demonstrates, these low and high-information environments differ substantially from the medium-information environment produced by the original design. This suggests that the AMCE estimate based on the conventional regression estimator may differ from the pAMCE s based on the two counterfactual distributions representing specific information environments of theoretical interest. The framework of the pAMCE is essential to systematically assess how the AMCE estimates might change under different profile distributions.

4.1.1 Experimental Designs

We introduce three experimental designs; the joint population randomization design, the marginal population randomization design, and the mixed randomization design. While all experimental designs allow for the consistent estimation of the pAMCE, they differ in terms of data requirements and assumptions.

We begin with the joint population randomization design. In this design, researchers randomize profiles according to their target profile distribution.

Definition 4 Joint Population Randomization Design

(2) $$ \begin{align} && {{\mathrm{Pr}}^{\texttt{R}}}({\mathbf{T}}_{ik} = {\mathbf{t}}) = {{\mathrm{Pr}}^{\ast}}({\mathbf{T}}_{ik} = {\mathbf{t}})\ \ \ \ \mathrm{for\ all }\ {\mathbf{t}} \in \mathrm{\ support\ of}\ {\mathbf{T}}_{ik}\ \ \mathrm{and\ for\ all\ } i\ \mathrm{ and }\ k, \end{align} $$

where ${\mathrm{Pr}^{\texttt {R}}}(\cdot )$ denotes the distribution used for randomization and ${\mathrm{Pr}^{\ast }}(\cdot )$ represents the target profile distribution.

This design is simple and intuitive since it directly incorporates the target profile distribution into randomization. The main advantage is that the design allows for nonparametric estimation of the pAMCE using a weighted difference-in-means estimator described in the next section.

While the joint population randomization design enables nonparametric estimation, it requires the knowledge of the joint distribution of profile attributes. In practice, this requirement might be difficult to satisfy for many applications. An alternative design that relaxes this stringent data requirement is the marginal population randomization design. Under this design, researchers randomize each factor independently according to its marginal profile distribution of the target population.

Definition 5 Marginal Population Randomization Design

(3) $$ \begin{align} && {{\mathrm{Pr}}^{\texttt{R}}}(T_{ijk\ell} = t) = {{\mathrm{Pr}}^{\ast}}(T_{ijk\ell} = t) \ \ \ \ \mathrm{for\ all\ levels}\ t \ \ \mathrm{and\ for\ all\ }\ i, j, k, \ell. \end{align} $$

For example, we randomize three factors {Gender, Race, Education} independently with each marginal distribution, ${\mathrm{Pr}^{\ast }}(\texttt {Gender}), {\mathrm{Pr}^{\ast }}(\texttt {Race}),$ and ${\mathrm{Pr}^{\ast }}(\texttt {Education})$ , respectively, rather than using the joint distribution ${\mathrm{Pr}^{\ast }}(\texttt {Gender}, \texttt {Race}, \texttt {Education})$ .

The main advantage of this approach is that it only requires information about separate marginals of the target profile distribution. Gathering data on marginal distributions is likely to be easier in most contexts. In fact, some researchers have begun to incorporate marginal distributions of the target profile population in their research (see Leeper and Robison, Reference Leeper and Robison2018). Another significant benefit is that we can estimate the pAMCE using simple difference-in-means under this design. In practice, this means that researchers can estimate the pAMCE using a linear regression because factors are independent of each other.

The marginal population randomization design is not free of limitations. In particular, without further assumptions, this design estimates the approximate pAMCE where we only partially capture the target profile distribution. Nevertheless, compared to the uAMCE, this design already greatly improves the external validity of conjoint analysis. Indeed, a similar approximation strategy is often used in other contexts, including survey research, in which sampling weights are computed using population marginals, and causal inference with observational data, in which observed covariates are balanced only with respect to their marginal means.

What assumption is required for the consistent estimation of the pAMCE only with separate marginal distributions rather than the joint distribution of profile attributes? It turns out that we only need to assume the absence of three-way or higher order interactions among factors. Suppose that there are three factors Gender, Race, and Education, and they have two-way interactions; Gender $\,\times\,$ Race, Gender $\,\times\,$ Education, and Race $\,\times\, $ Education. In this case, a simple difference-in-means estimator is still consistent for the pAMCE so long as there exists no three-way or higher order interaction such as Gender $\,\times\, $ Race $\,\times\, $ Education. It is important to emphasize that the marginal population randomization design allows for the existence of any two-way interaction, which often captures the strongest interaction in many applications.

There are several ways to address concerns about the assumption of no three-way or higher-order interaction. First, researchers can extend this marginal population randomization design by incorporating the partial joint distributions. Suppose that the joint distribution ${\mathrm{Pr}^{\ast }}(\texttt {Race}, \texttt {Education})$ is available while all other factors are randomized independently according to their separate marginal distributions. In this case, we can consistently estimate the pAMCE of Gender via a weighted difference-in-means (see Section 4.1.2) even when there exists the three-way interaction Gender $\,\times\, $ Race $\,\times\, $ Education if the joint distribution of Race and Education is incorporated into randomization. In general, if we incorporate the joint distributions of M factors, the consistent estimation of the pAMCE is possible even if there exist $(M+1)$ -way interactions involving these factors. Finally, we can test the assumption of no three-way and higher-order interactions using the standard F-test (see Section 4.2.4).

As the final design, we introduce the mixed randomization design, which can yield more efficient estimates when researchers are interested in only a small number of factors (e.g., one or two) and view the remaining factors as background information they control for. For this design, we first separate L factors into two types ${\mathbf {T}} = \{{{\mathbf {T}}^{\mathcal {M}}}, {{\mathbf {T}}^{\mathcal {C}}}\}$ ; (1) main factors of interest ${{\mathbf {T}}^{\mathcal {M}}}$ , for which researchers wish to estimate the pAMCE, and (2) control factors $\;{{\mathbf {T}}^{\mathcal {C}}}$ , which are included as the background information. The distinction between the main and control factors is essential because there is a statistical tradeoff; as the number of the main factors increases, the estimation of the pAMCE becomes less precise. Under the mixed randomization design, we randomize the main factors of interest based on the uniform distribution and the control factors based on their target profile distribution.

Definition 6 Mixed Randomization Design

(4) $$ \begin{align} \mathrm{Main\ factors:} \ \ \ \ & {{\mathrm{Pr}}^{\texttt{R}}}(T_{ijk\ell} = t) = \frac{1}{D_{\ell}} \ \ \ \ \mathrm{for\ all\ levels}\ t\ \mathrm{in\ factor\ } \ell \in \mathcal{M} \ \ \mathrm{and\ for\ all }\ i, j, k\notag\\\mathrm{Control\ factors:} \ \ \ \ & {{\mathrm{Pr}}^{\texttt{R}}}({{\mathbf{T}}^{\mathcal{C}}}_{ik} = {\mathbf{t}}) = {{\mathrm{Pr}}^{\ast}}({{\mathbf{T}}^{\mathcal{C}}}_{ik} = {\mathbf{t}})\ \ \ \ \mathrm{for\ all\ } i\ \mathrm{ and }\ k. \end{align} $$

For example, as in the original study (Ono and Burden, Reference Ono and Burden2019), suppose researchers are primarily interested in estimating the pAMCEs of factor Gender and use the other 12 factors as control factors. Under the mixed design, we randomize Gender using uniform while randomizing the remaining factors based on their target profile distribution.

This design has two primary advantages. First, by prespecifying a small number of main factors at the design stage, researchers can increase the research transparency and credibility in the same way that preregistration does (Blair et al., Reference Blair, Cooper, Coppock and Humphreys2019). Second, under the mixed randomization design, we can often estimate the pAMCEs of the main factors more efficiently than under the two alternative designs. In fact, we show that when researchers have a single main factor, the mixed randomization design is optimal under the assumption of no cross-profile interaction effects (see Supplemental Appendix D.3). In contrast, when multiple factors are of interest, the comparison of statistical efficiency across the three designs gives an inconclusive answer (see Section 4.1.3 for the sample size formula).

4.1.2 The Weighted Difference-in-Means Estimator

We introduce a general weighted difference-in-estimator that is consistent for the pAMCE under all three experimental designs described above. We then show how this general estimator can be simplified under some designs.

Formally, the weighted difference-in-means estimator of the pAMCE can be written as follows (Hájek, Reference Hájek, V. P. and D. A.1971),

(5) $$ \begin{align} \widehat{\tau}^{\ast}_{\ell} (t_1, t_0) \ = \ \frac{\sum_{i=1}^N \sum_{j=1}^J \sum_{k=1}^K {\mathbf{1}}\{T_{ijk\ell} = t_1\} w_{ijk\ell} Y_{ijk}}{\sum_{i=1}^N\sum_{j=1}^J \sum_{k=1}^K {\mathbf{1}}\{T_{ijk\ell} = t_1\} w_{ijk\ell}} - \frac{\sum_{i=1}^N \sum_{j=1}^J \sum_{k=1}^K {\mathbf{1}}\{T_{ijk\ell} = t_0\} w_{ijk\ell} Y_{ijk}}{\sum_{i=1}^N \sum_{j=1}^J \sum_{k=1}^K {\mathbf{1}}\{T_{ijk\ell} = t_0\} w_{ijk\ell}}, \end{align} $$

where the weights are defined as,

(6) $$ \begin{align} w_{ijk\ell} & \ = \ \frac{1}{{{\mathrm{Pr}}^{\texttt{R}}}(T_{ijk\ell} \mid {\mathbf{T}}_{ijk,-\ell}, {\mathbf{T}}_{i,-j,k})} \times \frac{{{\mathrm{Pr}}^{\ast}}({\mathbf{T}}_{ijk,-\ell}, {\mathbf{T}}_{i,-j,k})}{{{\mathrm{Pr}}^{\texttt{R}}}({\mathbf{T}}_{ijk,-\ell}, {\mathbf{T}}_{i,-j,k})}. \end{align} $$

The weights equal the product of two terms. The first term represents the randomization distribution of $T_{ijk\ell }$ given all the other factors $\{{\mathbf {T}}_{ijk, -\ell }, {\mathbf {T}}_{i, -j, k}\}$ , whereas the second term is the ratio between the target profile distribution of $\{{\mathbf {T}}_{ijk, -\ell }, {\mathbf {T}}_{i, -j, k}\}$ and their randomization distribution. Therefore, the weights are greater for observations that are more prevalent in the target profile distribution than in the randomization distribution. We prove the consistency of this estimator in Supplemental Appendix D.1.

Under the joint population randomization design, the second term of the weights is equal to one and thus, weights are simplified as follows,

$$ \begin{align*} w^{\texttt{Joint}}_{ijk\ell} & \ = \ \frac{1}{{{\mathrm{Pr}}^{\ast}}(T_{ijk\ell} \mid {\mathbf{T}}_{ijk,-\ell}, {\mathbf{T}}_{i,-j,k})}. \end{align*} $$

Under the marginal population randomization design, both the first and second terms are canceled out and hence, weights are equal to one for all observations. Therefore, simple difference-in-means is consistent for the pAMCE under the assumption of no three-way or higher-order interaction.

Result 1 (Estimation under Marginal Population Randomization Design)

Under the assumption of no three-way or higher-order interaction, the following simple difference-in-means estimator is consistent for the pAMCE after randomizing profiles according to the marginal population randomization design (Equation (3)).

(7) $$ \begin{align} \frac{\sum_{i=1}^N \sum_{j=1}^J \sum_{k=1}^K {\mathbf{1}}\{T_{ijk\ell} = t_1\} Y_{ijk}}{\sum_{i=1}^N\sum_{j=1}^J \sum_{k=1}^K {\mathbf{1}}\{T_{ijk\ell} = t_1\} } - \frac{\sum_{i=1}^N \sum_{j=1}^J \sum_{k=1}^K {\mathbf{1}}\{T_{ijk\ell} = t_0\} Y_{ijk}}{\sum_{i=1}^N \sum_{j=1}^J \sum_{k=1}^K {\mathbf{1}}\{T_{ijk\ell} = t_0\} } \xrightarrow{p} \tau^{\ast}_{\ell} (t_1, t_0) \end{align} $$

This difference-in-means estimator can be computed by regressing $Y_{ijk}$ on an intercept and ${\mathbf {X}}_{ijk\ell }$ with regression where ${\mathbf {X}}_{ijk\ell }$ is a vector of $(D_\ell - 1)$ dummy variables for the levels of $T_{ijk\ell }$ excluding the baseline level $t_0.$ Then, this difference-in-means estimator equals the estimated coefficient on the dummy variable for the level $t_1$ of factor $\ell $ (Greene, Reference Greene2011; Hainmueller et al., Reference Hainmueller, Hopkins and Yamamoto2014). We provide the proof in Supplemental Appendix D.2.

Finally, under the mixed randomization design, while weights do not have a simple expression, we can use the general weighted difference-in-means estimator given in Equation (5).

In practice, the proposed weighted difference-in-means estimator can be computed via a weighted linear regression model.Footnote ⁴ Since the weighted linear regression is used only to compute the nonparametric weighted difference-in-means estimator, no additional modeling assumption is imposed. This weighted regression estimator generalizes the regression estimator proposed in Hainmueller et al. (Reference Hainmueller, Hopkins and Yamamoto2014).

4.1.3 Effective Sample Size

When using the proposed weighting estimator, it is important to compute the effective sample size (ESS) to determine the statistical efficiency of each design prior to conducting an experiment. We use Monte Carlo simulation by randomizing profiles according to a specific design and then compute the ESS as follows (Kish, Reference Kish1965),

(8) $$ \begin{align} \mbox{ESS} \ = \ \frac{(\sum_{ijk} w_{ijk\ell})^2}{\sum_{ijk} w_{ijk\ell}^2}. \end{align} $$

When weights are equal to one for every observation, the ESS is equal to the total sample size $NJK$ . As weights diverge from one, the ESS becomes smaller. Using ESS, we can easily compute the following standard error multiplier between any two designs,

(9) $$ \begin{align} \sqrt{\frac{\mbox{ESS under one design}}{\mbox{ESS under another design}}}, \end{align} $$

which quantifies the expected ratio of standard error that would result under one design over that under another design. By computing the ESS and the standard error multiplier at the design stage, researchers can choose an experimental design that most efficiently estimates the pAMCEs. Note that since weights are different for each pAMCE, we must compute these statistics separately.

4.2.1 Latent Utility Model

We begin by introducing a latent utility model that allows all two-way interactions. Specifically, we assume that the latent utility for each profile is a function of the main effect of each factor, the two-way interactions between all the factors, and the two-way interaction of the same factor between the two profiles within a given pair (e.g., the effect of age of one profile may depend on the age of the other profile). The modeling assumption is violated if three-way or higher order interaction effects exist. Although we believe that in most practical settings this assumption approximately holds, we offer a simple model specification test in Section 4.2.4.

Formally, our latent utility model of respondent i for profile j when compared against profile $j^\prime $ in the kth task is defined as follows,

(10) $$ \begin{align} \widetilde{Y}_{ijk}({\mathbf{T}}_{ijk}, {\mathbf{T}}_{ij^\prime k}) \ = & \ \ \widetilde\alpha + \sum_{\ell = 1}^L {\mathbf{X}}_{ijk\ell}^\top \widetilde{\beta}_\ell + \sum_{\ell = 1}^L \sum_{\ell^\prime \neq \ell} ({\mathbf{X}}_{ijk\ell} \times {\mathbf{X}}_{ijk\ell^\prime})^\top \widetilde{\gamma}_{\ell\ell^\prime} - \sum_{\ell = 1}^L {\mathbf{X}}_{ij^\prime k\ell}^\top \widetilde{\beta}_\ell \notag \\ & \ - \sum_{\ell = 1}^L \sum_{\ell^\prime \neq \ell} ({\mathbf{X}}_{ij^\prime k\ell} \times {\mathbf{X}}_{ij^\prime k\ell^\prime})^\top \widetilde{\gamma}_{\ell\ell^\prime} + \sum_{\ell = 1}^L ({\mathbf{X}}_{ij k\ell}\times {\mathbf{X}}_{ij^\prime k\ell})^\top \widetilde{\delta}_{\ell \ell} + \widetilde\epsilon_{ijk}, \end{align} $$

where ${\mathbf {X}}_{ijk\ell }$ is a vector of $(D_\ell - 1)$ dummy variables for the levels of $T_{ijk\ell }$ excluding the baseline level and $\,\times\, $ represents the cartesian product operator, for example, $({\mathbf {X}}_{ijk\ell } \times {\mathbf {X}}_{ijk\ell ^\prime })^\top \widetilde {\gamma }_{\ell \ell ^\prime } = \sum _{d=1}^{D_{\ell }-1} \sum _{d^\prime =1}^{D_{\ell ^\prime }-1} X_{ijk\ell d} X_{ijk\ell ^\prime d^\prime } \widetilde {\gamma }_{\ell d \ell ^\prime d^\prime }$ . The coefficients $\widetilde \beta _\ell $ denote the main effects of factor $\ell $ , while the coefficients $\widetilde \gamma _{\ell \ell ^\prime }$ indicate two-way interactions between the two factors $\ell $ and $\ell ^\prime $ . Finally, the coefficients $\widetilde \delta _{\ell \ell }$ represent two-way interactions between factor $\ell $ across the two profiles j and $j^\prime $ . Under the assumption of no profile-order effects, the effects of factors in profile j and those in profile $j^\prime $ are symmetric. This is why the effect of ${\mathbf {X}}_{ijk\ell }$ is $\widetilde \beta _{\ell }$ and that of ${\mathbf {X}}_{ij^\prime k\ell }$ is $-\widetilde \beta _{\ell }$ . Similarly, the effect of ${\mathbf {X}}_{ijk\ell } \times {\mathbf {X}}_{ijk\ell ^\prime }$ is $\widetilde \gamma _{\ell \ell ^\prime }$ while that of ${\mathbf {X}}_{ij^\prime k\ell } \times {\mathbf {X}}_{ij^\prime k\ell ^\prime }$ is $-\widetilde \gamma _{\ell \ell ^\prime }$ .

As in the conventional latent utility model, we do not directly observe the latent utility. Instead, we observe the choices made by respondents. Each respondent is assumed to choose profile j when its latent utility is higher than the latent utility of the other profile $j^\prime $ , that is,

$$ \begin{align*} Y_{ik} ({\mathbf{T}}_{ijk}, {\mathbf{T}}_{ij^\prime k}) \ = \ \begin{cases} 1 & \ \mbox{if } \ \ \widetilde{Y}_{ijk}({\mathbf{T}}_{ijk}, {\mathbf{T}}_{ij^\prime k})> \widetilde{Y}_{ij^\prime k}({\mathbf{T}}_{ijk}, {\mathbf{T}}_{i j^\prime k}), \\ 0 & \ \mbox{otherwise.} \end{cases} \end{align*} $$

There are many ways to connect the latent utility model to the choice outcome model. For example, when we assume the error term follows the type I extreme value distribution, we obtain the well-known conditional logit model (McFadden, Reference McFadden and Zarembka1974). For the ease of interpretation, we rely on the following linear probability model (Egami and Imai, Reference Egami and Imai2019),

(11) $$ \begin{align} & \ \mathrm{Pr}(Y_{ik} = 1 \mid {\mathbf{T}}_{ijk}, {\mathbf{T}}_{ij^\prime k}) \notag \\ \ = & \ \bigg\{\widetilde{Y}_{ijk}({\mathbf{T}}_{ijk}, {\mathbf{T}}_{ij^\prime k}) - \widetilde{Y}_{ij^\prime k}({\mathbf{T}}_{ijk}, {\mathbf{T}}_{ij^\prime k})\bigg\} + 0.5 \notag \\ \ = & \ \alpha + \sum_{\ell = 1}^L ({\mathbf{X}}_{ijk\ell} - {\mathbf{X}}_{ij^\prime k\ell})^\top\beta_\ell + \sum_{\ell = 1}^L \sum_{\ell^\prime \neq \ell} ({\mathbf{X}}_{ijk\ell} \times {\mathbf{X}}_{ijk\ell^\prime}-{\mathbf{X}}_{ij^\prime k\ell} \times {\mathbf{X}}_{ij^\prime k\ell^\prime})^\top\gamma_{\ell\ell^\prime} + \sum_{\ell = 1}^L ({\mathbf{X}}_{ij k\ell}\times {\mathbf{X}}_{ij^\prime k\ell})^\top\delta_{\ell \ell} \end{align} $$

where the coefficients have direct connections to the latent utility model given in Equation (10), that is, $\beta _\ell = 2\widetilde {\beta }_\ell $ , $\gamma _{\ell \ell ^\prime } = 2\widetilde {\gamma }_{\ell \ell ^\prime }$ and $\delta _{\ell \ell } = 2\widetilde {\delta }_{\ell \ell }.$ We estimate this linear probability model via ordinary least squares by regressing $Y_{ijk}$ on an intercept, the difference in the main terms for all the factors, the difference in the interaction terms for all the two-way interactions, and the interaction terms across profiles for all the factors.

This model does not impose the linearity assumption because each level of a given factor enters the model as a separate dummy variable. The model also allows for all two-way interactions between and across profiles. Therefore, the key assumption is the absence of three-way or higher order interactions, which can be easily relaxed at the expense of statistical efficiency.

4.2.2 Estimation of the Population AMCE

Using the above linear probability model, we can estimate the pAMCE as a weighted average of the estimated coefficients,

(12) $$ \begin{align} \widehat{\tau}^{\ast}_{\ell} (t_1, t_0) & \ = \ \widehat{\beta}_{\ell 1} + \sum_{\ell^\prime \neq \ell} \sum_{d = 1}^{D_{\ell^\prime}-1}\widehat{\gamma}_{\ell 1 \ell^\prime d} {{\mathrm{Pr}}^{\ast}}(T_{ijk\ell^\prime}=d) + \sum_{d = 1}^{D_{\ell}-1} \widehat{\delta}_{\ell 1 \ell d} {{\mathrm{Pr}}^{\ast}}(T_{ijk\ell}=d). \end{align} $$

where the marginal distributions are used as weights. Thus, under the two-way interactive linear probability model, we only need to collect the marginal distributions of the target profile population ${\mathrm{Pr}^{\ast }}(T_{ijk\ell }=d)$ . This greatly relaxes data requirements in practice.

As we saw earlier, when there is no interaction between or across factors, the uAMCE equals the pAMCE. That is, when $\widehat {\gamma }_{\ell 1 \ell ^\prime d} = \widehat {\delta }_{\ell 1 \ell d} = 0$ , we have $\widehat {\tau }^{\ast }_{\ell } (t_1, t_0) = \widehat {\tau }^{\texttt {U}}_{\ell } (t_1, t_0) = \widehat {\beta }_{\ell 1}.$ In addition, it is straightforward to estimate the difference between the uAMCE and the pAMCE,

$$ \begin{align*} \widehat{\mbox{Diff}} \ \ = \ \ & \widehat{\tau}^{\ast}_{\ell} (t_1, t_0) - \widehat{\tau}^{\texttt{U}}_{\ell} (t_1, t_0) \\ = \ \ & \sum_{\ell^\prime \neq \ell} \sum_{d = 1}^{D_{\ell^\prime}-1}\widehat{\gamma}_{\ell 1 \ell^\prime d} \{{{\mathrm{Pr}}^{\ast}}(T_{ijk\ell^\prime}=d) - {{\mathrm{Pr}}^{\texttt{U}}}(T_{ijk\ell^\prime}=d)\}+ \sum_{d = 1}^{D_{\ell}-1} \widehat{\delta}_{\ell 1 \ell d} \{{{\mathrm{Pr}}^{\ast}}(T_{ijk\ell}=d) - {{\mathrm{Pr}}^{\texttt{U}}}(T_{ijk\ell}=d)\}. \end{align*} $$

Thus, as mentioned earlier, the difference is large when there exist significant interactions and when the target profile distribution is far away from the uniform distribution. Finally, we can decompose this difference as the sum of components due to different factors,

(13) $$ \begin{align} \widehat{\mbox{Diff}} \ = \ \sum_{\ell^\prime = 1}^L \widehat{\mbox{Diff}}_{\ell^\prime} \ = \ \sum_{\ell^\prime = 1}^L \sum_{d = 1}^{D_{\ell^\prime}-1}\widehat{\gamma}_{\ell 1 \ell^\prime d} \{{{\mathrm{Pr}}^{\ast}}(T_{ijk\ell^\prime}=d) - {{\mathrm{Pr}}^{\texttt{U}}}(T_{ijk\ell^\prime}=d)\}. \end{align} $$

Through this decomposition, researchers can unpack the origin of the difference between the uAMCE and the pAMCE.

4.2.3 Regularization

The main drawback of the model-based exploratory analysis is its large estimation uncertainty. When there are many factors and each factor has several levels, the model with all two-way interaction effects can produce large standard errors. We consider regularization as a way to partially recoup this loss of statistical efficiency relative to the design-based confirmatory analysis. For example, the conjoint analysis of Ono and Burden (Reference Ono and Burden2019) contains 13 factors with a total of 49 levels. This means that the estimated pAMCE will be the weighted average of a large number of interaction terms. In such cases, a regularized regression approach can be effective in reducing estimation uncertainty.

In particular, we follow Egami and Imai (Reference Egami and Imai2019) and collapse levels within factors using a regularized regression. For instance, even though Ono and Burden (Reference Ono and Burden2019) use six levels for factor Age (36, 44, 52, 60, 68, 76 years old), not all the differences between the six levels may be relevant. It may be, for example, that the effects for the first three levels are indistinguishable from each other and can be collapsed into fewer levels (e.g., 36/44/52, 60/68, 76 years old). We can use a regularized regression to identify such coarsening patterns. By reducing the number of levels, the proposed regularized regression can improve efficiency and estimate the pAMCE more precisely.

Specifically, we estimate the pAMCE by collapsing levels while avoiding regularization bias through cross fitting (Chernozhukov et al., Reference Chernozhukov2018). We begin by randomly splitting data into two parts, training and test data. Using the training data, we first collapse levels within factors via the generalized lasso (Tibshirani and Taylor, Reference Tibshirani and Taylor2011),

(14)

where we select tuning parameter $\lambda $ using cross validation. By weighting according to effect size, the adaptive weights help regularize smaller effects more and larger effects less.Footnote ⁵ Importantly, we do not shrink the coefficients $\beta _{\ell d}$ themselves and instead regularize their differences $|\beta _{\ell d} - \beta _{\ell ,d-1}|$ so that we can collapse unnecessary levels (Egami and Imai, Reference Egami and Imai2019). When levels are unordered, researchers can use an alternative penalty that regularizes all pairwise differences, that is, $\sum _{\ell = 1}^L \sum _{d=0}^{D_\ell -1}\sum _{d^\prime \neq d}|\beta _{\ell d} - \beta _{\ell ,d^\prime }|.$

Second, using the separate test data, we fit the proposed linear probability model with collapsed levels and then estimate the pAMCE based on the weighted average expression given in Equation (12). Because unnecessary levels are removed in the previous step, we can estimate the pAMCE more precisely. It is important that we collapse levels with the training data and estimate the pAMCE with the separate test data to remove bias due to regularization.

Finally, we flip the role of training and test data and repeat the two steps described above. We average the two estimates from each test data as the estimate of the pAMCE. For uncertainty estimates, we use the block bootstrap by sampling respondents with replacement. We implement the cross-fitting for each bootstrap replicate. Uncertainty estimates are calculated based on the empirical distribution of the estimated pAMCE over the bootstrap sample. In Supplemental Appendix F, we provide simulation studies to show how much the proposed regularization method can improve efficiency without inducing bias.

4.2.4 Assessing the Absence of Higher-Order Interaction

The model introduced above (Equation (11)) as well as the marginal population randomization design (Equation (3)) assumes the absence of three-way or higher order interaction. We can directly test the assumed absence of three-way interaction by conducting the standard F-test. Specifically, we incorporate three-way interactions between three factors $\ell , \ell ^\prime ,$ and $\ell ^{\prime \prime }$ by adding $({\mathbf {X}}_{ijk\ell } \times {\mathbf {X}}_{ijk\ell ^\prime } \times {\mathbf {X}}_{ijk\ell ^{\prime \prime }})^\top \zeta _{\ell \ell ^\prime \ell ^{\prime \prime }}$ to the two-way interactive model of Equation (11) where $\zeta _{\ell \ell ^\prime \ell ^{\prime \prime }}$ is a vector of coefficients for the three-way interactions. Then, we test the existence of this three-way interaction via F-test with the null hypothesis, $H_0: \zeta _{\ell \ell ^\prime \ell ^{\prime \prime }} = \mathbf {0}$ . When the statistical power of detecting three-way interaction effects is of concern, it is recommended to rely on the regularization approach described above.

5.1.1 Design-Based Analysis

We begin by performing the design-based confirmatory analysis proposed in Section 4.1. Because in the original study each attribute is randomized according to the uniform distribution, we conduct a simulation study to assess the performance of the marginal population randomization and mixed randomization designs.Footnote ⁶ To do this, we first fit a linear regression model with all two-way interactions between the thirteen factors summarized in Table 1 and use this estimated model as the true data generating process. For the marginal population randomization design, we randomize each factor independently based on a marginal distribution of the target population. For the mixed randomization design, the primary factor of interest, that is, gender, is randomized according to the uniform distribution and the remaining factors are randomized using their target population distribution. We estimate the pAMCE via the simple difference-in-means under the marginal population design and the weighted difference-in-means under the mixed design. Standard errors are clustered by respondents. We repeat the same procedure 100 times and average over point estimates and standard errors.

The left plot in the top row of Figure 4 presents the results. First, we focus on the results based on the mixed randomization design. In contrast to the estimates of the uAMCE, we find that the pAMCE is estimated to be $-2.20$ percentage points (95% confidence interval $=[-3.30, -1.10]$ ) when using the distribution of Republicans and $2.64$ percentage points ( $[1.53, 3.74]$ ) for Democrats. We obtain similar results under the marginal population randomization design although the standard errors are slightly larger. Recall that under the uniform distribution, the estimated uAMCE of gender on vote choice is small and negative. This demonstrates that the estimated AMCE critically depends on the target distribution of candidates’ attributes.

One key conclusion of the original study is that the negative effect of being female is found only for presidential candidates but not for congressional candidates. We revisit this finding by using the real-world politicians as the target profile distributions. In particular, we now conduct the design-based confirmatory analysis separately for congressional and presidential candidates. These results are presented in the top row of the second and third columns of Figure 4. Consistent with the original analysis, the estimated uAMCE of being female is $-0.09$ percentage points ( $[-1.71, 1.48]$ ) for congressional candidates and $-2.42$ percentage points ( $[-3.96, -0.88]$ ) for presidential candidates. For presidential candidates (the third plot in the top row), the pAMCE of being female is similar to the corresponding uAMCE for both Democratic and Republican distributions. Female presidential candidates face barriers compared to male candidates regardless of party. This result shows that the pAMCE and the uAMCE estimates can be similar even when the target profile distribution is far from uniform. This is because there exists little interaction between gender and other factors within this subgroup (see formal discussions in Section 3.3).

Interestingly, for congressional candidates (the second plot in the top row), the results of the uAMCE and pAMCEs diverge. The uAMCE implies that gender has little effect in congressional races. Yet a more realistic profile distribution suggests a more nuanced finding: women are disadvantaged when they run as Republicans and advantaged when they run as Democrats. Under the mixed randomization design, female Republican candidates are $1.98$ percentage points ( $[0.42, 3.54]$ ) less likely to be chosen than their male counterparts, while female Democratic candidates are $5.69$ percentage points ( $[4.13, 7.25]$ ) more likely to be chosen. The latter effect is large in substantive terms, equaling the effect of candidates’ experience in office and their position on deficit reduction.

5.1.2 Model-Based Analysis

Now, we illustrate the model-based exploratory analysis introduced in Section 4.2. This approach is useful especially when researchers are interested in exploring the pAMCE with conjoint experiments that have already been conducted using the uniform or any distributions different from the target distribution. First, we focus on estimating the pAMCE for both presidential and congressional candidates together as done in the original analysis. As explained in Section 4.2.3, we incorporate all two-way interactions among all the thirteen factors in Table 1 except for Office and then collapse levels within factors using the generalized lasso. Standard errors are calculated with 2,000 block bootstraps clustered by respondents.

As expected, the results are similar to those from the design-based analysis but with larger standard errors (see the left plot in the bottom row of Figure 4). The estimated pAMCE is $-2.87$ percentage points ( $[-6.39, 0.63]$ ) when using the distribution of Republican politicians and $2.84$ percentage points ( $[-0.20, 5.87]$ ) for Democratic politicians. We also repeat the same analysis for presidential and congressional candidates separately (the second and third plots in the second row). As in the design-based confirmatory analysis, we find that female congressional candidates have a disadvantage when they run as Republicans and have an advantage when they run as Democrats.

The standard errors are much larger than those in the design-based confirmatory analysis because the uniform distribution used in the experiment is markedly different from the target profile distribution. This postadjustment of the large differences reduces the effective sample size. Since the model-based analysis marginalizes all the two-way interactions, the efficiency loss is especially severe when the number of factors and levels within each factor are large, as in this example. Although regularization recoups some of this efficiency loss, the design-based analysis yields smaller standard errors in such high dimensional designs.

To investigate the sources of the difference between the uAMCE and pAMCE, we apply the decomposition formula given in Equation (13). For the sake of illustration, we focus on the difference between the estimated uAMCE and pAMCE for congressional Democratic candidates (the second plot in the bottom row). The first plot of Figure 5 shows the results of this decomposition, with each row representing the difference attributable to a single factor. We find, for example, that the difference due to factor Security Policy is about $1.6$ percentage points ( $[0.14, 3.12]$ ), implying that the estimated AMCE increases by $1.6$ percentage points when we use the distribution of Democratic politicians for Security Policy rather than the uniform distribution. Importantly, less than 20% of the overall difference is attributable to Party, meaning that we cannot estimate the pAMCE just by considering the interaction between Gender and Party. The results show that the difference between the uAMCE ( $-0.09$ percentage points) and pAMCE ( $6.17$ percentage points) can be attributed to a combination of many factors even though the contribution of each factor is not necessarily precisely estimated. Even if the difference due to each factor is small, when aggregated across many factors, the overall difference between the uAMCE and the pAMCE can be substantial. This result underscores the need to collect relevant data for as many factors as possible when building the target distribution.

Figure 5 Decomposing the difference between the estimated uAMCE and pAMCE of being a female candidate. For congressional candidates, we compare the uAMCE and the pAMCE based on the Democrat distribution. The first plot decomposes the overall difference into each factor. The second and third factors investigates how effect heterogeneity and the difference in the profile distributions result in the difference in the uAMCE and the pAMCE.

To further unpack the source of this difference, we examine the conditional average marginal component effect (cAMCE). The cAMCE is the AMCE of the factor of interest—in this case, gender—conditional on the level of another factor.Footnote ⁷ A difference in the cAMCEs across the levels of the second factor implies an interaction with the factor of interest. For example, a difference in the cAMCEs of Gender conditional on Security Policy would indicate an interaction between Gender and Security Policy. We can use the cAMCE to determine whether each factor’s contribution to the difference between the uAMCE and the pAMCE (the first plot of Figure 5) is due to large interactions, large changes in distribution, or a combination of the two. If interactions between the primary factor of interest and secondary factors are responsible for most of the difference between the uAMCE and the pAMCE, even small changes in distribution will make the uAMCE different from the pAMCE.

The right two plots of Figure 5 visualize the distributions of three factors Security Policy, Abortion Policy, and Party alongside the cAMCEs of Gender conditional on each of the three factors. For example, the first row in the third plot presents the estimated cAMCE of being female relative to male, conditional on having the Security Policy factor equal to Cut military budget. Focusing on the Security Policy factor, we observe that although its cAMCEs are modest in size, the distribution for Democratic politicians differs substantially from the uniform distribution. Thus, the difference induced by the Security Policy factor is being driven primarily by distributional differences. Repeating this approach for each factor tells us whether the difference between the uAMCE and the pAMCE is a function of distributional changes or causal interactions.

As an important diagnostic, we evaluate the assumption of no three-way interactions. In particular, we incorporate three-way interactions between Gender, Party, and each of the four policy positions given that the difference between the uAMCE and the pAMCE is mostly attributable to those factors. Because we have information about the joint distribution of politicians’ attributes, we use them when we marginalize over three-way interactions to estimate the pAMCE. Figure 6 shows that the pAMCE estimates based on the three-way interaction model are similar to those based on the two-way model both in terms of point estimates and standard errors. This result demonstrates that, even when researchers have access only to marginal distributions of the target profile population, it is possible to consistently estimate the pAMCEs by using the marginal population randomization design.

Figure 6 Assess the existence of three-way interactions. We compare the pAMCE estimates from models that assume two-way interactions and that incorporate three-way interactions.

Finally, we examine the robustness of the pAMCE estimates based on the 115th Congress to alternative profile distributions based on the available candidate-level data rather than the data on elected politicians. Although these candidate-level data do not contain information for all factors, we can take into account a number of important candidates’ characteristics. In particular, we use DIME data set (Bonica, Reference Bonica2015) and the Reflective Democracy (RefDem) datasetFootnote ⁸ to obtain the profile distributions of three key variables (race, gender, and experience). We also use our substantive knowledge to investigate different theoretically relevant profile distributions on policy dimensions. We provide details of these alternative profile distributions in Supplemental Appendix C. We show that the pAMCE estimates are robust to these different profile distributions that more accurately reflect the real-world distribution of political candidates (see Figure A3 in Supplemental Appendix C). These results imply that the difference between the pAMCE and uAMCE is mainly driven by the fact that the uniform distribution is far away from the actual distribution of politicians’ characteristics. In contrast, the difference between the distribution of attributes for elected politicians and that for candidates is relatively minor and has little impact on the empirical findings.

5.2.1 Design-Based Analysis

We begin with the design-based confirmatory analysis. To estimate the pAMCE, we use the marginal population randomization design by randomizing each factor according to the counterfactual distributions of interest. We also employ the mixed randomization design, retaining the uniform distribution for a primary factor of interest—copartisanship, in this case—and using the counterfactual distributions for all remaining factors. We rely on a weighted difference-in-means estimator (Equation (5)) and cluster standard errors by respondents.

The left plot of Figure 7 presents results of this analysis. Consistent with the original finding, the pAMCE of copartisanship is estimated to be the largest under the low information distribution ( $61.84$ percentage points, $[59.06, 64.62]$ ) while the effect is the smallest under the high information distribution ( $38.39$ percentage points, $[35.13, 41.65]$ ) using the mixed randomization design. Results are similar under the marginal population randomization design. Thus, the importance of copartisanship in subjects’ voting decisions can vary by more than $20$ percentage points depending on the information environment.

Figure 7 The estimated population AMCEs of copartisanship in Peterson (Reference Peterson2017). We estimate the pAMCE of being copartisan under three different distributions – a medium information distribution and the low and high information distributions.

5.2.2 Model-Based Analysis

Next, we estimate the same three quantities using model-based exploratory analysis. To do so, we run an unregularized linear regression using all two-way interactions between the ten factors described in Table 2. While regularization is generally preferred, the factors here are binary. Since the goal of regularization is to improve efficiency by collapsing levels of a factor that have similar effects, regularization is not needed in this case. Standard errors are based on 2,000 block bootstraps clustered by respondents.

The second plot in Figure 7 presents these results. As in the design-based confirmatory analysis, the pAMCE of copartisanship is the largest under the low information distribution ( $61.87$ percentage points, $[57.34, 66.40]$ ) and the effect is the smallest under the high information distribution ( $38.21$ percentage points, $[33.69, 42.73]$ ). Although standard errors for the model-based exploratory analysis are larger than those of the design-based confirmatory analysis, the difference between them in this application is relatively small. This is due to the fact that the design in Peterson (Reference Peterson2017) is low-dimensional, comprised only of binary factors.

After showing that copartisanship effects are indeed smaller when a larger number of candidate characteristics are shown, the author conducts the second analysis to unpack the mechanism by identifying which information is responsible for reducing the effect of copartisanship. To answer this question, he considers an extreme counterfactual distribution, in which only one factor (in addition to copartisanship) is shown to respondents and examines the difference in the pAMCE of copartisanship with and without this additional factor. The author repeats this analysis separately for each of the nine factors and finds that policy positions on spending and abortion result in the largest differences.

In our pAMCE framework, there is no need to consider each factor in isolation. Instead, we directly examine the sources of the difference in the pAMCE of copartisanship between the low and high information environments. To do so, we use the decomposition formula. The left plot of Figure 8 shows that the difference observed in Figure 7 is mainly driven by two factors, Spending Stance ( $-7.60$ percentage points, $[-12.16, -3.04]$ ) and Abortion Stance ( $-10.77$ percentage points, $[-15.38, -6.16]$ ). This result suggests that respondents use copartisanship mainly as a cue for policy stances on spending and abortion, consistent with the original findings.

Figure 8 Decomposition of the difference between the two pAMCEs of copartisanship between high and low information environment. The left plot decomposes the overall difference into each factor. The difference is mainly due to the two factors, Spending Stance and Abortion Stance. The second and third plots investigate how the conditional AMCE and the difference in the profile distributions contribute to the difference.

Finally, we examine why these factors drive the difference in the copartisanship effect between the low and high information environments. The second and third plots of Figure 8 present the distribution and the cAMCEs of each factor. Taking Spending Stance as an illustration, we find that the cAMCE for Shown (the bottom estimate) is much smaller than for Not Shown (the second estimate from the bottom). There is a strong interaction between factors Party and Spending Stance, yielding the large difference of the pAMCE between the high and low information environments. In contrast, little difference exists in the cAMCEs of copartisanship conditional on Age (see the first and second estimates in the third plot). This is why the difference of the pAMCE due to Age is small (third estimate in the first plot).