In this essay, we closely examine three aspects of the Reporting Guidelines for this journal, as described by Gerber et al. (2014, Journal of Experimental Political Science 1(1): 81–98) in the inaugural issue of the Journal of Experimental Political Science. These include manipulation checks and when the reporting of response rates is appropriate. The third, most critical, issue concerns the committee's recommendations for detecting errors in randomization. This is an area where there is evidence of widespread confusion about experimental methods throughout our major journals. Given that a goal of the Journal of Experimental Political Science is promoting best practices and a better understanding of experimental methods across the discipline, we recommend changes to the Standards that will allow the journal to play a leading role in correcting these misunderstandings.

Three Canadian research studies which used similar sampling strategies are examined. Data on their survey experiences are presented so as to assist those planning research to do so more efficiently. The response and completion rates of the three studies were quite similar, suggesting that some predictions might be made for similarly-designed future studies. The findings may give heart to those wondering whether their response and refusal rates are adequate or embarassing. Detailed reporting of survey research sampling experiences and adoption of a uniform terminology is called for.

The NHLBI Twin Study is a longitudinal study of cardiovascular disease risk factors in 514 pairs of white, middle aged, male, veteran twins. The initial examination took place between 1969-1973. Ten years later, 81% of the living cohort returned for a second examination. Data collected up to 30 years prior to recruitment for the initial examination were used to characterize participants and nonparticipants; data from the initial examination were used to characterize returnees and nonreturnees to the second examination. Participants had significantly lower diastolic blood pressure and higher socioeconomic status than nonparticipants as measured thirty years earlier. Between the first and second examinations, the mortality of participants was less than 50% of the mortality of nonparticipants. Returnees to the second examination had a better health profile at the initial examination than nonreturnees, with significantly lower levels of cigarette smoking, glucose intolerance, hypertension, and diabetes and higher levels of pulmonary function. However, returnees were more obese than nonreturnees. Thus, this study of cardiovascular disease risk factors in twins appears to be affected by response bias in a way similar to studies of individuals. Additional analyses of biases that may affect the genetic component of the study indicated that factors related to classical twin analyses were relatively unaffected by selection.