2.1. For Memory

I call “alternative hypothesis” (H_A) the statistical hypothesis that is experimentally tested and that is accepted when the null hypothesis is rejected. The significance level (a.k.a. α level) is the largest value a p-value may have if the null hypothesis (H ₀) is to be rejected. It is often set at .05 in experimental psychology. A negative result is any experimental result that does not permit the rejection of the null hypothesis because the p-value of the relevant statistic is larger than the significance level.

The power of a test is the probability of rejecting the null hypothesis if the null hypothesis is false and if a particular point alternative hypothesis is true.Footnote ¹ Thus, 1 minus the power of a test is the probability that this test will commit a type II error when this point alternative hypothesis is true (failing to accept the alternative hypothesis when it is true—this probability is sometimes called β). The power of a test is influenced by the population effect size, the significance level, and the sample size. The larger the sample size, the higher the power of a test. The higher the significance level, the higher the power of a test. The larger the population effect size, the higher the power of a test.

Statisticians distinguish between observed power and prospective power. Power is said to be “observed” when one uses various descriptive statistics (e.g., the pooled sample variance or the difference between the sample means) to estimate the corresponding parameters of the populations from which the experimental samples are drawn. For instance, one could estimate the variances of the populations from which the samples are drawn by means of the variances of the samples themselves. One then computes the probability of rejecting the null hypothesis if samples of a particular size are drawn from these populations, assuming a particular significance level. By contrast, power is said to be “prospective” when one stipulates, instead of estimating, the values for the parameters of the populations from which the relevant samples are drawn. One then computes the probability of rejecting the null hypothesis that would obtain if the samples were drawn from populations with these parameter values. Prospective power is regularly used in planning experiments (but see Sedlmeier and Gigerenzer Reference Sedlmeier and Gigerenzer1989).

2.2. Negative Results, Null Hypotheses, and Power

Jacob Cohen proposed that a null hypothesis can be inferred from a negative result if and only if the test in which a negative result was obtained has a high power—that is, if and only if the test was such that the null hypothesis would have been rejected if it were false. More precisely, he writes (Reference Cohen1988, 16):

Thus, Cohen is proposing the following inferential procedure:

The justification of this procedure is straightforward. Cohen notes that there is a parallelism between null hypothesis significance testing and the procedure just described. In the case of null hypothesis significance testing, one decides to reject the null hypothesis and to accept the alternative hypothesis when the p-value is smaller than the significance level. By doing so, one takes the risk of committing a type I error, but this risk is small, and in the long run one will not make more than a small number of type I errors if one follows this procedure consistently. In the case of the power-based inferential procedure, one decides to accept the null hypothesis and to reject the alternative hypothesis when and only when the power of the test is larger than a high threshold (say, .95).Footnote ⁴ By doing so, one takes the risk of committing a type II error, but this risk is small, and in the long run one will not make more than a small number of type II errors if one follows this procedure consistently.Footnote ⁵ If one takes null hypothesis significance testing to be justified because of its property of controlling long-run error rates, then by parity of reasoning one should hold the power-based inferential procedure to be justified too.

2.3. Examples

Psychologists rarely publish negative results, and as a consequence they have few occasions to infer the null hypothesis on the basis of the power of a test. Even in those relatively rare articles that report negative results, psychologists too often satisfy themselves with an informal analysis. For instance, Hare and colleagues (Reference Hare, Melis, Woods, Hastings and Wrangham2007) compared chimpanzees’ and bonobos’ willingness to collaborate to obtain some food (experiment 2). Participants (either two chimpanzees or two bonobos) were paired; each participant had to pull a rope at the same time as the other participant in order to bring food toward itself. When the food was dispersed in a way that allowed each participant to benefit from the product of its contribution to the collaborative effort (i.e., some food) on its own, chimpanzees and bonobos collaborated equally. Inferring the null hypothesis from a negative result, Hare and colleagues write that “in support of the emotional-reactivity hypothesis, there was no difference between the species’ ability to cooperate spontaneously to obtain divisible-dispersed food” (620). Power is not reported either in the main article or in the online supplementary data. No alternative formal analysis is presented.

Some articles do appeal to power to infer the null hypothesis from a negative result, although the power-based inferential procedure is not always rigorously applied. In his report of failed replications of Boroditsky's (Reference Boroditsky2001) famous study about the cultural diversity of the mental representation of time, Chen (Reference Chen2007) estimated the population effect size by means of the effect size found in the study and, on this basis, computed the sample size needed to reach a power of .80, assuming a significance level of .05. While taking power into account is commendable, this analysis does not follow the power-based inferential procedure very closely. In another report about Boroditsky (Reference Boroditsky2001), January and Kako (Reference January and Kako2007) also mentioned power, but they too failed to follow the power-based inferential procedure closely.

3.1. Observed Power 1

There are many ways to compute the observed power of an experiment, depending on the number of relevant population parameters that are estimated by means of the sample statistics. One could first estimate all the relevant population parameters. For instance, for a t-test, one could estimate the mean and variance of the population by means of the mean and variance of the sample drawn from it.

The problem with this approach is that, so computed, observed power is uninformative since there is then a one-to-one relationship between p-values and power. As a result, knowing the power of an experiment does not add any information to knowing the p-value. If one would not infer the null hypothesis from a negative result on the basis of the p-value of the relevant statistic alone (and, for obvious reasons, one should not do this), one should not infer from the observed power, so computed, either.

3.2. Observed Power 2

Of course, one need not estimate all the relevant population parameters by means of the corresponding sample statistics. Instead, one could estimate some population parameters (e.g., the variance of the population from which the sample in a t-test is drawn) by means of the sample statistics and stipulate the values of the others (e.g., the mean of this population) in order to compute the power of the test. In this case, there is not a one-to-one relation between the power of the test and the obtained p-value, and the problem just discussed disappears.

However, so computed, observed power gives rise to the following paradox. Experiments E ₁ and E ₂ are two (one-sample) experiments with the following properties. The sample sizes are the same (); the means observed in E ₁, m ₁, and E ₂, m ₂, are equal (); and the standard deviation obtained in E ₁, s ₁, is smaller than the standard deviation obtained in E ₂, s ₂ (). Furthermore, both E ₁ and E ₂ yield nonsignificant findings, that is, negative results.

Now, because for a one-sample t-test, t is larger in E ₁ than in E ₂. Hence, the probability of obtaining a statistic of this size or a larger one if the null hypothesis is true is larger in E ₁ than in E ₂. That is, because , the p-value is lower in E ₁ than in E ₂ (). This point can be made intuitive as follows. Consider, for example, the two histograms in figures 1 and 2 corresponding to E ₁ and E ₂, respectively. The mean, mode, and median are the same in these two histograms, but the data are more spread out in E ₂ than in E ₁. Because the variance in E ₂ is larger than the variance in E ₁, the fact that the sample mean differs from 0 is less likely to be due to sampling in E ₁ than in E ₂. Thus, the sample in E ₂ is more plausibly drawn from a population whose mean is 0 than is the sample in E ₁.

Figure 1. Histogram for E ₁.

Figure 2. Histogram for E ₂.

Consider now the observed power of E ₁ and E ₂. Suppose that the null hypothesis is false and that the populations from which the two samples are drawn have the same mean, which is larger than 0 (). Variables s ₁ and s ₂ are used to estimate the standard deviations of these populations. For any stipulated population mean, E ₁ is more likely to reject the null hypothesis than is E ₂ since less of the mass is close to zero in the case of E ₁. Figures 3 and 4 illustrate this point. Figure 3 represents the population from which the sample in E ₁ is drawn, and figure 4, the population from which the sample in E ₂ is drawn. Because a smaller part of the area under the curve is close to 0 in figure 3 than in figure 4, one is more likely to reject the null hypothesis by sampling randomly from the population represented in figure 3 than from the population represented in figure 4. That is, the power of E ₁ is larger than the power of E ₂.

Figure 3. Population in E ₁.

Figure 4. Population in E ₂.

We end up with a paradox. Because , E ₁ provides more evidence against the null hypothesis and for the alternative hypothesis than E ₂. Because the power of E ₁ is larger than the power of E ₂, E ₁ also provides more evidence for the null hypothesis and against the alternative hypothesis than E ₂.

3.3. Prospective Power

It may be tempting to suggest that the paradox results from the use of observed power rather than prospective power to infer the null hypothesis from a negative result. This would be a mistake, however: it is simple to derive a paradox similar to the one just presented when one uses prospective power instead of observed power.

Experiments E ₁ and E ₂ are two (one-sample) experiments such that , , and . Furthermore, both E ₁ and E ₂ yield negative results. To determine the power of E ₁ and E ₂, we stipulate that the two samples are drawn from the same population, whose mean is different from 0. Because the samples are drawn from an identical population and because the sample sizes are the same, the prospective power of E ₁ and E ₂ is the same. As a result, they provide the same amount of evidence for the null hypothesis and against the alternative hypothesis. However, because , the p-value is smaller in E ₁ than in E ₂, and E ₁ provides more evidence against the null hypothesis and for the alternative hypothesis than E ₂.

3.4. Conclusion

However power is computed, some pairs of experiments are such that, if it is appropriate to use power to infer the null hypothesis from a negative result, the first experiment seems to provide more evidence both for and against the null hypothesis than the second experiment (or more evidence against the null hypothesis than the second experiment and the same amount of evidence for it as the second experiment). It seems natural to conclude that it is inappropriate to use power to infer the null hypothesis from a negative result. This is indeed the moral that Hoenig and Heisey draw (Reference Hoenig and Heisey2001, 5): “Power calculations tell us how well we might be able to characterize nature in the future given a particular state and statistical study design, but they cannot use information in the data to tell us about the likely states of nature. With traditional frequentist statistics, this is best achieved with confidence intervals, appropriate choices of null hypotheses, and equivalence testing.”

4.1. Two Interpretations of p-Values

The p-values can be understood in two different ways. First, one can assume that a p-value measures the strength of evidence against the null hypothesis. The smaller the p-value, the more evidence an experiment provides against the null hypothesis and for the alternative hypothesis. If the p-value in a first experiment is .005 and the p-value in a second experiment is .01, then the first experiment provides more evidence against the null hypothesis than the second. For Kempthorne and Folks (Reference Kempthorne and Folks1971, 314), for instance, a p-value is “a way of producing a quantification of strength of evidence.”

Alternatively, one can assume that a p-value is a property of the data that allows for the application of a decision rule specifying when the null hypothesis is to be rejected and the alternative hypothesis accepted: reject the null hypothesis and accept the alternative hypothesis when and only when the probability of observing a statistic of this size or a more extreme one (e.g., a t-score) is lower than the significance level (say, .05). Under this interpretation, p-values are not taken to measure the strength of evidence against the null hypothesis. One does not treat the fact that the p-value in a first experiment is .005 and the p-value in a second experiment is .01 as meaning that the first experiment provides more evidence against the null hypothesis. Rather, one treats the two experiments identically: in both cases, one simply rejects the null hypothesis and accepts the alternative hypothesis.

4.2. Two Interpretations of Power

The two interpretations of p-values have counterparts in the case of power. First, one can assume that the power of an experiment measures the strength of evidence for the null hypothesis. Second, one can assume that the power of an experiment is a property of a test that allows for the application of a decision rule specifying when the null hypothesis is to be accepted and the alternative hypothesis rejected: accept the null hypothesis from a negative result when and only when power is above some threshold. Noticeably, Cohen assumes this second interpretation in the quotation given in section 2, and I have followed him when I presented the rule governing the inference of null hypotheses from negative results.

4.3. Avoiding the Paradoxes

In the paradoxes described in section 3, p-values and power are treated as measures of evidence. One compares how much evidence for and against the null hypothesis E ₁ and E ₂ provide, and paradoxes emerge because E ₁ provides more evidence both for and against the null hypothesis than does E ₂ (or more evidence against the null hypothesis than does E ₂ and the same amount of evidence for it as E ₂).

However, if one endorses the alternative interpretations of p-values and power, then there is no experiment such that the antecedents of the rule about rejecting the null hypothesis on the basis of a low p-value and of the rule about accepting the null hypothesis on the basis of a high power are simultaneously satisfied. The former rule only applies when the p-value is below the significance level, while the second rule only applies when it is above the significance level (i.e., when a negative result occurs). As a result, in no experiment do these two rules result in contradictory decisions: accept and reject the null hypothesis. Thus, the paradoxes are avoided when p-values and power are treated not as measures of evidence but as properties of the data and test that allow for the application of decision rules.

One may argue that to circumvent the paradoxes discussed above it is sufficient to merely avoid interpreting power as a measure of evidence for the null hypothesis, while still interpreting p-values as measures of evidence. However, first, it is unclear what could justify the resulting asymmetry between power and p-values. Second, variants of the paradoxes discussed earlier that speak against this proposal are easy to formulate. Consider the case of observed power discussed in section 3.2: E ₁ and E ₂ are two (one-sample) experiments such that , , and . Furthermore, both E ₁ and E ₂ yield negative findings. As we have seen, in this case, , and the power of E ₁ is larger than the power of E ₂. It is possible that the power of E ₁, but not the power of E ₂, is above the threshold required for accepting the null hypothesis and rejecting the alternative hypothesis (e.g., .95). In this case, we would accept the null hypothesis in E ₁ but not in E ₂, even though E ₁ provides more evidence against the null hypothesis than does E ₂ (since p-values are interpreted as measures of evidence).

4.4. Moral

The moral that should be drawn from the paradoxes discussed in section 3 is not that, if one tests hypotheses by means of null hypothesis significance testing, one should not use power to infer a null hypothesis from a negative result, as one could have initially thought. What the paradoxes reveal is that one should not treat power and p-values as measures of evidence but rather as properties of the data and test that allow for the application of decision rules for rejecting and accepting hypotheses. When one does so, one can use the power of an experiment to infer the null hypothesis from a negative result.

5.1. The Proposed Solution to the Paradoxes Is Arbitrary

One could wonder why it is preferable to conclude from the paradoxes discussed in section 3 that p-values and power should not be treated as measures of evidence rather than to conclude that power should not be used to infer the null hypothesis from a negative result. One reason for embracing the first alternative is that interpreting p-values as measures of evidence gives rise to other problems. For instance, if p-values are measures of evidence, two experiments yielding identical p-values provide the same amount of evidence, even if their sample sizes are different (Cornfield's [Reference Cornfield1966] α postulate). This is inconsistent with treating the likelihood ratio as a measure of evidence since this ratio is influenced by the sample size (Royall Reference Royall1986). It is worth noting briefly that, in contrast to other arguments against the interpretation of p-values as measures of evidence (e.g., Royall Reference Royall1997), the argument advanced here does not appeal to, and thus is not dependent on, potentially controversial principles about the nature of evidence.

5.2. The Proposed Solution Leaves No Room for the Notion of Evidence

One could reject the proposed solution on the grounds that it leaves no room for measuring the strength of the evidence for statistical hypotheses. Since this short article is obviously not the place to discuss the role of the notion of evidence in statistical inference, I only note that philosophers should not view as patently absurd the idea that the acceptance of statistical hypotheses on the basis of statistics is not to be interpreted in evidential terms: after all, this interpretation was favored by Jerzy Neyman, whose methods are widely used throughout the sciences (Machery Reference Machery2012).

5.3. The Proposed Solution Is in Tension with Some Common Practices and Norms in Psychology

Philosophers of science should plausibly hold scientists’ practices as well as the norms they accept as prima facie correct. On this basis, one could object to the recommendation to avoid treating p-values and power as measures of evidence that it goes against both the usual practices of psychologists and the norms put forward by professional organizations in psychology such as the American Psychological Association (APA).

Psychologists often seem to hold that the smaller the p-value obtained in a test, the more confident one can be in rejecting the null hypothesis. Experimental research on psychologists’ understanding of p-values supports this claim: when psychologists are asked how confident they are in the rejection of the null hypothesis, their confidence decreases with increasing p-values (e.g., Poitevineau and Lecoutre Reference Poitevineau and Lecoutre2001).Footnote ⁶ It is natural to interpret these findings as follows: psychologists view p-values as measures of the evidence against the null hypothesis, and their confidence in the rejection of the null hypothesis decreases with the diminishing available evidence against it.

Furthermore, some official norms in psychology seem to assume an evidential interpretation of p-values. In its influential report, the APA Task Force on Statistical Inference recommended that psychologists report the exact p-values instead of reporting whether p-values are below the significance level, as used to be common: “It is hard to imagine a situation in which a dichotomous accept-reject decision is better than reporting an actual p value or, better still, a confidence interval” (Wilkinson and the Task Force on Statistical Inference Reference Wilkinson1999, 599). It is natural to conclude from this recommendation that the APA Task Force views p-values as measures of evidence. When they are so treated, the exact value of p matters: a p-value equal to, say, .03 provides more evidence against the null hypothesis than a p-value equal to, say, .04. By contrast, if p-values are just properties of the data that allow for the application of a rule for rejecting the null hypothesis, it does not matter whether a p-value is equal to .03 or .04, and reporting the exact value is of no particular use.

There is thus little doubt that the solution to the paradoxes proposed in this article is at odds with some common practices and norms in psychology. The disagreement with psychologists’ practices should not carry much normative weight, however, since their understanding of p-values and of the foundations of their statistical tests is often poor (e.g., Oakes Reference Oakes1986). The same is true of the disagreement with the norms in psychology. The norms put forward by professional organizations are not entirely consistent with one another (see Fidler [Reference Fidler2002, 756] on the discrepancies about p-values between the report by the APA task force and the statistical section of the fifth edition of the APA Publication Manual). Furthermore, focusing only on the report of the APA task force, this report leaves psychologists without proper norms for inferring null hypotheses from negative results, a striking shortcoming. Finally, what norms of statistical inference psychology should adopt remains an area of active debate among psychologists and applied statisticians.