Optimal treatment regimes (OTRs) have been widely employed in computer science and personalized medicine to provide data-driven, optimal recommendations to individuals. However, previous research on OTRs has primarily focused on settings that are independent and identically distributed, with little attention given to the unique characteristics of educational settings, where students are nested within schools and there are hierarchical dependencies. The goal of this study is to propose a framework for designing OTRs from multisite randomized trials, a commonly used experimental design in education and psychology to evaluate educational programs. We investigate modifications to popular OTR methods, specifically Q-learning and weighting methods, in order to improve their performance in multisite randomized trials. A total of 12 modifications, 6 for Q-learning and 6 for weighting, are proposed by utilizing different multilevel models, moderators, and augmentations. Simulation studies reveal that all Q-learning modifications improve performance in multisite randomized trials and the modifications that incorporate random treatment effects show the most promise in handling cluster-level moderators. Among weighting methods, the modification that incorporates cluster dummies into moderator variables and augmentation terms performs best across simulation conditions. The proposed modifications are demonstrated through an application to estimate an OTR of conditional cash transfer programs using a multisite randomized trial in Colombia to maximize educational attainment.

The use of cluster robust standard errors (CRSE) is common as data are often collected from units, such as cities, states or countries, with multiple observations per unit. There is considerable discussion of how best to estimate standard errors and confidence intervals when using CRSE (Harden 2011; Imbens and Kolesár 2016; MacKinnon and Webb 2017; Esarey and Menger 2019). Extensive simulations in this literature and here show that CRSE seriously underestimate coefficient standard errors and their associated confidence intervals, particularly with a small number of clusters and when there is little within cluster variation in the explanatory variables. These same simulations show that a method developed here provides more reliable estimates of coefficient standard errors. They underestimate confidence intervals for tests of individual and sets of coefficients in extreme conditions, but by far less than do CRSE. Simulations also show that this method produces more accurate standard error and confidence interval estimates than bootstrapping, which is often recommended as an alternative to CRSE.

This letter deals with a very simple question: if we have grouped data with a binary-dependent variable and want to include fixed effects in the specification, can we meaningfully compare results using a linear model to those estimated with a logit? The reason to doubt such a comparison is that the linear specification appears to keep all observations, whereas the logit drops the groups where the dependent variable is either all zeros or all ones. This letter demonstrates that a linear specification averages the estimates for all the homogeneous outcome groups (which, by definition, all have slope coefficients of zero) with the slope coefficients for the groups with a mix of zeros and ones. The correct comparison of the linear to logit form is to only look at groups with some variation in the dependent variable. Researchers using the linear specification are urged to report results for all groups and for the subset of groups where the dependent variable varies. The interpretation of the difference between these two results depends upon assumptions which cannot be empirically assessed.

U.S. state politics researchers often analyze data with observations grouped into clusters. This structure commonly produces unmodeled correlation within clusters, leading to downward bias in the standard errors of regression coefficients. Estimating robust cluster standard errors (RCSE) is a common approach to correcting this bias. However, despite their frequent use, recent work indicates that RCSE can also be biased downward. Here the author provides evidence of that bias and offers a potential solution. Through Monte Carlo simulation of an ordinary least squares (OLS) regression model, the author compares conventional standard error (OLS-SE) and RCSE performance to that of a bootstrap method that resamples clusters of observations (BCSE). The author shows that both OLS-SE and RCSE are biased downward, with OLS-SE being the most biased. In contrast, BCSE are not biased and consistently outperform the other two methods. The author concludes with three replications from recent work and offers recommendations to researchers.

Objectives: This work has investigated under what conditions cost-effectiveness data from a cluster randomized trial (CRT) are suitable for analysis using a cluster-adjusted nonparametric bootstrap. The bootstrap's main advantages are in dealing with skewed data and its ability to take correlations between costs and effects into account. However, there are known theoretical problems with a commonly used cluster bootstrap procedure, and the practical implications of these require investigation.

Methods: Simulations were used to estimate the coverage of confidence intervals around incremental cost-effectiveness ratios from CRTs using two bootstrap methods.

Results: The bootstrap gave excessively narrow confidence intervals, but there was evidence to suggest that, when the number of clusters per treatment arm exceeded 24, it might give acceptable results. The method that resampled individuals as well as clusters did not perform well when cost and effectiveness data were correlated.

Conclusions: If economic data from such trials are to be analyzed adequately, then there is a need for further investigations of more complex bootstrap procedures. Similarly, further research is required on methods such as the net benefit approach.