Conservatism of the response criterion

N&S use the study by Hassin et al. (Reference Hassin, Ferguson, Shidlovski and Gross2007) to illustrate the response bias problem. In this particular study, subjects were asked to indicate if a masked stimulus was an Israeli flag or a control stimulus (scrambled flag). In such a “yes/no” task, subjects might adopt a conservative decision criterion and respond “no” even on those occasions in which they had a fleeting glimpse of the real flag. While this is a real concern, it is easily addressed. One solution is to calculate a bias-free measure (d′), which is a standard technique in signal detection theory (Green & Swets Reference Green and Swets1966). Another solution is to use a two-alternative forced-choice (2AFC) paradigm, where subjects are provided with both possible stimuli simultaneously and their task is to indicate which one of the two had been presented immediately before as the masked stimulus. The advantage of using the 2AFC paradigm is that it (1) greatly discourages response biases and (2) generally provides a more sensitive measure of stimulus detection and/or discriminability (Macmillan & Creelman Reference Macmillan and Creelman2004. There are now several reports of subliminal perception in the literature that have used the 2AFC paradigm to assess subjects' awareness of the masked stimuli (cf. Dell'Acqua & Grainger Reference Dell'Acqua and Grainger1999; Finkbeiner Reference Finkbeiner2011; Finkbeiner & Palermo Reference Finkbeiner and Palermo2009). Thus, the response-conservatism artifact is not an intractable problem, and so claims of subliminal perception affecting decision making should not be dismissed simply because this is a well-known problem.

Effect of task difficulty on motivation

A somewhat more difficult problem to address is the task-difficulty artifact (Pratte & Rouder Reference Pratte and Rouder2009). In the standard subliminal perception experiment, subjects are first asked to classify the targets and, subsequently, the masked stimuli. Evidence in favor of subliminal perception depends upon (1) a reliable behavioral effect in the target-classification task and (2) chance-level performance in the prime-classification task. A concern with this two-task design is that the prime-classification task is, by design, very difficult and, hence, poor performance might be due to task difficulty, not subliminality of the primes. If so, then the prime-classification task will underestimate subjects' awareness of the prime stimuli. This is a well-known problem in the subliminal perception literature that has become known as the “task-difficulty artifact” (Pratte & Rouder Reference Pratte and Rouder2009). But well-known problems are not necessarily intractable problems. In fact, researchers have long known about the task-difficulty artifact and have addressed it in various ways. The most frequently employed solution is to interleave long-duration primes, which are easier for subjects to classify, into the prime-classification task. That serves to make the prime-classification task easier overall and, hence, is thought to encourage subjects to “keep trying” at an otherwise difficult task (cf. Finkbeiner & Caramazza Reference Finkbeiner and Caramazza2008; Finkbeiner et al. Reference Finkbeiner, Forster, Nicol and Nakamura2004; Grainger et al. Reference Grainger, Diependaele, Spinelli, Ferrand and Farioli2003; Naccache & Dehaene Reference Naccache and Dehaene2001).

Recently, we sought to eliminate the task-difficulty artifact by bringing performance in the prime-classification task up to ceiling levels (>90% accurate), even while maintaining subliminality (Finkbeiner Reference Finkbeiner2011). We did this by presenting the masked prime stimuli (the words dog or may) in red or green ink. Immediately following presentation of the masked stimulus, we presented subjects with the four possible prime stimuli (formed by crossing the two prime words with the two ink colors) and asked them to point to the correct one. Using this task, we found that subjects were very good at color identification (>90%), but were at chance at word identification (~50%). Thus, the task-difficulty artifact is not an intractable problem, and so, again, claims of subliminal perception should not be dismissed simply because this is a well-known problem.

A third problem: null sensitivity

A far more pernicious problem, and one that N&S do not mention, is the null-sensitivity problem. Here the problem has to do with a lack of sufficient power for the standard null hypothesis significance test (NHST) approach to accurately resolve performance that is only slightly above chance levels. It may seem that the solution would be to increase the power of the experiment, but this is ultimately not feasible. For example, with a sample size of 21 subjects with a mean true performance of 52%, one would need approximately 570 prime-classification trials to bring the probability of wrongly accepting the null hypothesis down below 5%. With a true performance of 51%, approximately 2,300 trials would be needed. This is the null-sensitivity problem, and trying to resolve it through the addition of more and more trials is not practical for most researchers or their subjects.

Fortunately, the standard NHST approach is not the only way to test for chance-level performance. In a series of recent articles, Rouder et al. (Reference Rouder, Morey, Speckman and Pratte2007) and Morey et al. (Reference Morey, Rouder and Speckman2008; Reference Morey, Rouder and Speckman2009) have introduced a hierarchical model within the Bayesian framework that offers researchers a way to resolve the null-sensitivity problem. They have termed this the mass-at-chance (MAC) model. A virtue of MAC is that it penalizes underpowered designs, thereby mitigating the null-sensitivity problem. Small sample sizes yield highly variable posterior estimates of subjects' latent abilities, which makes it more difficult to claim subliminality (Rouder et al. Reference Rouder, Morey, Speckman and Pratte2007). In one recent study (Finkbeiner Reference Finkbeiner2011), we used the one-parameter MAC model (Morey et al. Reference Morey, Rouder and Speckman2008) and found that, with 9:1 odds in favor of subliminality, our experiment yielded 16:1 odds in favor of priming. Thus, even the most difficult problem in the subliminal perception literature, the null-sensitivity problem, is not without solutions.