4.3.1. Assuming ${\mathcal{H}}_{\mathcal{o}}$ vs assuming ${\mathcal{H}}_{\mathcal{a}}$#

When calculating the $p$-value for a test, we assume ${\mathcal{H}}_{\mathcal{o}}$ is true and then work out the probabliity that we might reject ${\mathcal{H}}_{\mathcal{o}}$ due to chance. In other words, we are working on the basis of a certain state of reality, assuming ${\mathcal{H}}_{\mathcal{o}}$ is true.

Assuming ${\mathcal{H}}_{\mathcal{o}}$ is true:

• $p$ is the proportion of false positives

• Specificity is $(1-p)$

In contrast, in power analysis, we consider the alternative reality in which ${\mathcal{H}}_{\mathcal{a}}$ is true. Power is then theh probability that we correctly reject the null.

Assuming ${\mathcal{H}}_{\mathcal{a}}$ is true:

• Power is (1-proportion of false negatives)

• Sensitivity = power

Terminology

Because the terms sensitivity and specificity are easy to mix up, I prefer to talk about:

• the $p$-value of a test (which is 1-specificity) and

• the power of a test (equivalent to sensitivity).

4.3.2. Criminal analogy#

It can help to think of an analogy to a criminal court:

• ${\mathcal{H}}_{\mathcal{o}}$: Everyone is innocent until proven guity

• ${\mathcal{H}}_{\mathcal{a}}$: He’s guilty, dammit!

In this analogy:

• A Type I error (false positive) is a wrongful conviction

• A Type II error (false negative) is when the criminal gets away due to insufficient evidence

4.3.3. Insufficient evidence#

The criminal analogy highlights the main cause of False Negatives (Type II errors) in scientific studies: insufficient evidence.

Often if we fail to get a significant result, when there is an underlying effect, it is because the sample size is too small.

This is because tests of statistical significance, and tests of statistical power, depend on three ingredients

• The effect size

• The sample size ($n$)

• The $\alpha$ value (the cut-off $p$-value for significance)

In general,

• The smaller the effect size, the larger the sample size neeed to detect it.

• The more stringent the $\alpha$ value, the larger the sample needed to detect an effect of a given size

  • $\alpha =0.001$ is more stringent than $\alpha =0.01$ etc

The next pages will cover the definition of effect size.

4.3.4. Sample size primarily affects power, not $p$-value#

Most people will accept the intuition that increasing the sample size should make an experiment ‘better’, more reliable or more believable. However it is worth understanding that increasing the sample size primarily affects the proportion of false negatives NOT false positives. Hence sample size is relevant for the power of a test more than its $p$-value. The reasons for this will be unpacked in the page Sample Size and Power in this chapter.