Difficulty: Intermediate | Prerequisites: Basic probability, familiarity with means and standard deviations.
Hypothesis testing is the core machinery of inferential statistics. It lets you move from "these two samples look different" to a quantified statement about whether the difference is likely real or just noise. This set of notes covers the type of test to choose (one-sided vs two-sided, paired vs unpaired), how to interpret the p-value that falls out, and what happens when you run many tests at once. If you are comfortable with sample means and variability but have not yet formally tested a hypothesis, start here.
A hypothesis test compares observed data against what you would expect under a null hypothesis (typically "no difference"). The p-value tells you how surprising your data would be if the null were true, and you reject the null when the p-value falls below your chosen significance level. When running multiple tests simultaneously, you need corrections like Bonferroni to avoid false positives.
Null hypothesis (H₀)
The default assumption that there is no effect or no difference between groups. In simple terms, it is the "nothing interesting is happening" position that you are trying to disprove.
Alternative hypothesis (H₁ or Hₐ)
The claim that there is a real effect or difference. This is what you are hoping to find evidence for. Think of it as the research question restated as a testable statement.
P-value
The probability of observing data at least as extreme as what you collected, assuming the null hypothesis is true. In simple terms, a small p-value means your data would be very unusual if there really were no effect.
Significance level (α)
The threshold below which you reject the null hypothesis, most commonly set at 0.05 (5%). Think of it as the line in the sand you draw before looking at the data.
Two-sided test (two-tailed test)
A test that checks for a difference in either direction, not just one. Use this when you have no prior reason to expect the effect to go a specific way. For instance, if you are comparing plant growth rates and either plant could plausibly grow faster, a two-sided test is appropriate.
One-sided test (one-tailed test)
A test that only checks for an effect in one specific direction. Use this only when you have a strong theoretical reason to expect the effect can only go one way.
Paired test
Used when each observation in one group has a natural match in the other group, such as before-and-after measurements on the same subjects.
Unpaired test (independent samples test)
Used when the two groups are made up of entirely different subjects with no one-to-one correspondence. For example, comparing the growth of two different species of plant grown in separate pots.
Bonferroni correction
A method for adjusting the significance threshold when you perform multiple hypothesis tests at once. You multiply each p-value by the number of tests (m), then compare the adjusted value to α. In simple terms, it makes your standard stricter to account for the increased chance of a false positive when you test many things simultaneously.
Statistical significance
A result is statistically significant when its p-value falls below the chosen significance level α. This does not mean the effect is large or important, only that it is unlikely to have arisen by chance alone.
One-sided vs two-sided: Default to a two-sided test unless you have a specific, pre-stated directional hypothesis. A two-sided test is more conservative and more widely accepted.
Paired vs unpaired: Ask whether each data point in group A has a natural partner in group B.
Same subjects measured twice (e.g. before and after treatment) → paired.
Different subjects in each group (e.g. two different plant species) → unpaired.
The p-value is not the probability that the null hypothesis is true. It is the probability of getting data this extreme if the null were true.
Compare the p-value to your pre-set significance level (α, usually 0.05).
p < α → reject H₀. The result is statistically significant.
p ≥ α → fail to reject H₀. You do not have sufficient evidence.
Example from the homework: a p-value of 0.0479 is less than 0.05, so you reject the null hypothesis that both plants grow at the same rate.
When you test many hypotheses at once, the probability of at least one false positive increases rapidly. If you run 20 independent tests at α = 0.05, you expect about one false positive even when nothing is really going on.
Multiply each raw p-value by the total number of tests (m).
Compare the adjusted p-value to α = 0.05.
Only reject the null for tests where the adjusted p-value remains below 0.05.
Example from the homework: with multiple metabolites tested, metabolites A and D survived correction (adjusted p < 0.05), while B (adjusted p = 0.4) and J (adjusted p = 9, effectively capped at 1) did not.
The Bonferroni correction is simple and widely used, but it is conservative. When the number of tests is very large, it can make it very hard to detect real effects.
Bonferroni adjusted p-value:
p_adjusted = p_raw × m
where m = total number of tests performed.
Reject H₀ when p_adjusted < α (typically 0.05).
Clinical trials routinely test a drug against multiple outcomes (blood pressure, cholesterol, mortality). Without a correction like Bonferroni, at least one outcome would likely appear "significant" by chance, leading to misleading conclusions about drug efficacy.
Genomics studies that scan thousands of genes for association with a disease use even stricter corrections (such as the Bonferroni threshold of p < 5 × 10⁻⁸ for genome-wide significance) for the same reason.
"p < 0.05 means the null hypothesis is wrong." It does not. It means the data are unlikely under H₀. The null could still be true; you may have observed an unusual sample.
"A p-value of 0.0479 is barely significant, so it must be a weak effect." Statistical significance and effect size are separate concepts. A tiny p-value can accompany a trivial effect in a large sample, and a large p-value can accompany a meaningful effect in a small sample.
"Failing to reject H₀ means H₀ is true." It means you did not find enough evidence to reject it. Absence of evidence is not evidence of absence.
"The comparison threshold for rejecting H₀ is 0.5." A common slip (and one that appears in the homework). The standard threshold is 0.05, not 0.5. Always double-check the decimal.
⚠️ You will almost certainly be asked to choose between a one-sided and two-sided test and to justify your choice.
⚠️ Know the difference between paired and unpaired tests. Misidentifying the pairing structure is a common exam error.
⚠️ Be able to apply the Bonferroni correction: multiply, compare, conclude. Expect a question that gives you a table of raw p-values and asks which remain significant after correction.
⚠️ Never write that p < 0.5 is sufficient to reject H₀. The threshold is 0.05.
True or false: A two-sided test checks for a difference in only one direction.
Fill in the blank: In an unpaired test, the subjects in each group are __________.
True or false: If you perform 10 tests, the Bonferroni correction multiplies each p-value by 10.
Fill in the blank: A p-value of 0.03 compared to α = 0.05 leads you to __________ the null hypothesis.
True or false: Failing to reject H₀ proves H₀ is true.
Answers: 1. False. 2. Different / independent. 3. True. 4. Reject. 5. False.
Q: You are comparing the growth rates of two different plant species grown in separate greenhouses. Should you use a paired or unpaired test, and why?
A: Unpaired, because the plants in each group are different subjects with no natural one-to-one pairing.
Q: Your two-sided t-test returns a p-value of 0.0479 at α = 0.05. State your conclusion about the null hypothesis.
A: Reject the null hypothesis. The p-value (0.0479) is less than the significance level (0.05), providing sufficient evidence that the groups differ.
Q: A researcher tests 10 metabolites for association with a disease. Metabolite X has a raw p-value of 0.04. After applying the Bonferroni correction, can the researcher claim statistical significance at α = 0.05?
A: No. The adjusted p-value is 0.04 × 10 = 0.4, which exceeds 0.05. The result is not significant after correction.
Q: Why is the Bonferroni correction considered conservative?
A: It reduces the chance of any false positive across all tests, but in doing so it also reduces statistical power, making it harder to detect real effects, especially when the number of tests is large.
This material connects directly to confidence intervals, which are the interval-estimation counterpart of hypothesis tests. A 95% confidence interval that excludes the null value corresponds to a two-sided test that rejects at α = 0.05.
Multiple testing corrections also appear in ANOVA post-hoc comparisons (Tukey, Scheffé), where you are comparing multiple group means and face the same inflation of false-positive risk.
hypothesis test, null hypothesis, alternative hypothesis, p-value, significance level, alpha, two-tailed test, one-tailed test, paired t-test, unpaired t-test, independent samples t-test, Bonferroni correction, multiple comparisons, multiple testing problem, family-wise error rate, FWER, Type I error, false positive, statistical significance, reject null, fail to reject