Claim 1: Linguists standardly ask a single informant about the acceptability of a few sentences (sect. 1.2, para. 2)

Claim 1 is a caricature of linguistic methodology that, to our knowledge, has never been supported by evidence. Nonetheless, a charitable interpretation of this claim reveals two separate concerns: (1) the routine use of small sample sizes, and (2) the susceptibility of AJs to investigator bias (Claim 2, below). An obvious consequence of using small samples sizes in research is an increase in errors (probably of all four types identified by Gelman & Carlin Reference Gelman and Carlin2014: I, II, Sign, and Magnitude). By performing a large-scale comparison of the published results in linguistics with retests of those results using large samples of naïve participants, one can evaluate the quality of their convergence rate. This cannot identify specific errors, but it can tell us whether the differences between methods actually produce different results.

Sprouse and Almeida (Reference Sprouse and Almeida2012) tested every English data point from a popular syntax textbook (Adger Reference Adger2003) using large samples of naïve participants. Out of 365 phenomena, they conservatively estimate a minimum convergence rate of 98%. Sprouse et al. (Reference Sprouse, Schütze and Almeida2013) randomly sampled 148 phenomena from a leading linguistics journal (Linguistic Inquiry), and conservatively estimate a convergence rate of 95% ( ±  5% because of the random sampling). These high (conservative) convergence rates suggest that the sample sizes used by linguists (whatever they are) historically have introduced little error to the empirical record for any combination of the following reasons: (1) the samples are larger than what critics claim; (2) the effect sizes are so large that small samples still yield good statistical power; or (3) AJ results are highly replicated before and after publication (e.g., Phillips Reference Phillips, Iwasaki, Hoji, Clancy and Sohn2009).

Claim 2: Acceptability judgments are highly susceptible to theoretical cognitive bias because linguists tend to use professional linguists as participants (sect. 1.2, para. 3)

This can also be addressed by the studies discussed above. Cognitive bias should predict sign reversals between naïve and expert populations. Sprouse and Almeida (Reference Sprouse and Almeida2012) found no sign reversals from the textbook data. Sprouse et al. (Reference Sprouse, Schütze and Almeida2013) reported a 1–3% sign reversal rate in the journal data. Mahowald et al. (Reference Mahowald, Graff, Hartman and Gibson2016a) and Häussler et al. (Reference Häussler, Juzek and Wasow2016) have replicated the latter without reporting an increased sign reversal rate (0–6%). Comparisons of naïve and expert populations also were conducted by Culbertson and Gross (Reference Culbertson and Gross2009), who report high inter- and intra-group correlations on 73 sentence types, and by Dąbrowska (Reference Dąbrowska2010). The latter found that, while experts gave less variable ratings than naïve participants on several sentence types, experts rated certain theoretically interesting syntactic violations as more acceptable than naïve participants, in apparent conflict with their theoretical commitments. Taken together, these results are not what one would expect if AJs were highly susceptible to cognitive bias.

Claim 3: Acceptability judgments are susceptible to differences in instructions (sect. 1.2, para. 3)

Claim 3 has been directly investigated by Cowart (Reference Cowart1997), who reports that the systematic manipulation of instructions does not change the pattern of acceptability judgments for factorial designs.

Claim 4: Acceptability judgments are impacted by sentence processing effects (sect. 1.2, para. 5)

Claim 4 is technically true, but B&P exaggerate its consequences. First, many classic lexical and sentence processing effects have relatively small or negligible effects on acceptability (e.g., Featherston Reference Featherston2009; Phillips Reference Phillips, Iwasaki, Hoji, Clancy and Sohn2009; Sprouse Reference Sprouse2008; Sprouse et al. Reference Sprouse, Wagers and Phillips2012). Second, very few syntactic phenomena have been proposed to be fully reducible to sentence processing effects. The lone exceptions to this appear to be constraints on long-distance dependencies (e.g., Kluender & Kutas Reference Kluender and Kutas1993; Hofmeister & Sag Reference Hofmeister and Sag2010), but in that case, a number of experimental studies have disproven the reductionist predictions (Phillips Reference Phillips2006; Sprouse et al. Reference Sprouse, Wagers and Phillips2012; Yoshida et al. Reference Yoshida, Kazanina, Pablos and Sturt2014). Thus, to the extent that AJs are impacted by sentence processing, it appears as though the effects can be dealt with like any other source of noise in an experimental setting.

Claim 5: Acceptability judgments reveal only set membership (sect. 1.2, para. 7)

Claim 5 is confusing. It is false in the sense that, if one is interested in set membership, this property still needs to be inferred from acceptability data, using a logic that maps that data type back to the relevant cognitive computations. In this, AJs are like any other data type in cognitive science: No data types, including priming, directly reveal the underlying computations of the human brain, and all data types require a linking hypothesis between the observable data and the unobservable cognitive process.

Claim 6: Acceptability judgments have yielded no consensus theory among linguists (sect. 1.2, para. 9)

Claim 6 is a strange criticism to make of any data type, especially AJs. First, the beliefs of scientists are a subjective issue based on how they weigh different kinds of evidence. Second, AJs are, by all accounts, a robust and replicable data type. Whatever disagreements there are in linguistics literature, they appear to obtain mostly at the level of interpreting, not establishing, the data (e.g., Phillips Reference Phillips, Iwasaki, Hoji, Clancy and Sohn2009).

In conclusion, we support B&P's desire to bring new evidence to bear on questions about linguistic representation. We caution, however, that advocacy for one method should not be bolstered by misleading comparisons, especially with methods such as AJs, which yield data that are demonstrably robust, highly replicable, and comparatively convenient and inexpensive to collect.