Difficulty: Intermediate | Prerequisites: Parts 1 and 2 of these notes (ANOVA Fundamentals; ANOVA Calculations and the F-Test).
A significant ANOVA result tells you that at least two group means differ, but it does not tell you which ones. This section covers what happens next: post-hoc (after-the-fact) procedures for identifying the specific pairs that differ. The central challenge is controlling error rates when making multiple comparisons. Tukey's procedure and the Bonferroni correction are the two methods emphasised in STAT 212, and both address the same underlying problem: the more comparisons you make, the more likely you are to find a "significant" result by chance alone.
Running many pairwise t-tests inflates the probability of at least one false positive (the experimentwise error rate). Tukey's procedure and the Bonferroni correction both keep this error rate under control. Tukey's builds confidence intervals for every pair of means using the Studentized Range distribution; Bonferroni simply divides α by the number of comparisons. Both are used after ANOVA rejects H₀.
Experimentwise Error Rate (EWER)
The probability of making at least one Type I error across a set of hypothesis tests. In plain terms, it is the chance that any of your comparisons gives a false positive, not just one of them.
Type I error
Rejecting a null hypothesis that is in fact true. In the ANOVA context, concluding that two means differ when they do not.
Pairwise comparison
A hypothesis test comparing the means of two specific groups. With k groups, there are k(k - 1)/2 possible pairs.
Bonferroni correction
A method that controls the EWER by dividing the desired overall significance level α by the number of comparisons m. Each individual test is then conducted at α/m.
Tukey's Procedure (Tukey's Honest Significant Difference)
A post-hoc method that constructs simultaneous confidence intervals for all pairwise differences in means, using the Studentized Range distribution (Q) to control the EWER at a specified level.
Studentized Range distribution (Q)
The sampling distribution used in Tukey's procedure. The critical value Q is looked up using k (number of groups) and dfE (error degrees of freedom) at the chosen significance level.
Confidence interval for µᵢ - µⱼ (Tukey's)
The interval (X̄ᵢ - X̄ⱼ) ± Q · √(MSE/2 · (1/nᵢ + 1/nⱼ)). If this interval does not contain zero, the two means are declared significantly different.
After a significant ANOVA with k groups, it is tempting to run pairwise t-tests for every combination of groups.
With k groups, there are k(k - 1)/2 pairwise comparisons. For k = 4, that is 6 comparisons; for k = 5, it is 10.
Each test carried out at α = 0.05 has a 5% chance of a Type I error individually. But across m independent tests, the probability of at least one Type I error is:
EWER = 1 - (1 - α)^m
For 10 tests at α = 0.05: EWER = 1 - (0.95)^10 ≈ 0.4013, meaning roughly a 40% chance of at least one false positive.
This inflation is why ANOVA exists in the first place (one omnibus test) and why specialised post-hoc procedures are needed for follow-up.
The simplest fix: if you want an overall EWER of α and you are making m comparisons, test each one at α/m.
For 10 comparisons with α = 0.05: adjusted significance level per test = 0.05 / 10 = 0.005.
Bonferroni is conservative (it makes it harder to reject), especially as m grows large. For many comparisons, Tukey's procedure is typically more powerful.
Choose the desired EWER (typically α = 0.05).
Look up the critical value Q from the Studentized Range table, using k (number of groups) and dfE (error degrees of freedom from the ANOVA).
Compute the confidence interval for each pair of means µᵢ - µⱼ:
(X̄ᵢ - X̄ⱼ) ± Q · √(MSE/2 · (1/nᵢ + 1/nⱼ))
When all group sizes are equal (nᵢ = nⱼ = n), this simplifies to: (X̄ᵢ - X̄ⱼ) ± Q · √(MSE/n)
Check whether each interval contains zero. If zero is not in the interval, conclude that µᵢ and µⱼ are significantly different at the chosen EWER.
Setup: k = 4 groups, N = 40, MSE = 12.5, n₁ = n₂ = 10, X̄₁ = 25, X̄₂ = 32, Q₀.₀₅,₄,₃₆ ≈ 3.81.
Difference: X̄₂ - X̄₁ = 32 - 25 = 7.
Margin of error: 3.81 · √(12.5/2 · (1/10 + 1/10)) = 3.81 · √(6.25 · 0.2) = 3.81 · √1.25 ≈ 3.81 · 1.118 ≈ 4.26.
Confidence interval: 7 ± 4.26 → [2.74, 11.26].
Since zero is not in [2.74, 11.26], conclude µ₁ and µ₂ are significantly different.
Tukey's is preferred when you want to compare all possible pairs of means. It is designed for this exact situation and is more powerful than Bonferroni for pairwise comparisons.
Bonferroni is more general and can be applied to any set of comparisons (not just all pairwise). It is simpler but more conservative.
Experimentwise Error Rate: EWER = 1 - (1 - α)^m
Bonferroni adjusted significance level: α_adjusted = α / m
Tukey's confidence interval for µᵢ - µⱼ: (X̄ᵢ - X̄ⱼ) ± Q_(α, k, dfE) · √(MSE/2 · (1/nᵢ + 1/nⱼ))
Equal sample sizes version: (X̄ᵢ - X̄ⱼ) ± Q_(α, k, dfE) · √(MSE/n)
Post-hoc comparisons are where ANOVA results become actionable. A clinical trial comparing four treatments might find a significant overall F-test, but the hospital needs to know which treatment outperforms which. Tukey's procedure answers that question while keeping the false-positive risk at the stated level across all comparisons simultaneously. Bonferroni corrections appear routinely in genomics, neuroimaging, and any field that runs thousands of tests at once.
Students often think that rejecting H₀ in ANOVA tells you which specific means differ. It does not. ANOVA is an omnibus test; you need a post-hoc procedure to identify specific pairs.
A common error in Tukey's calculation is forgetting the 1/2 inside the square root. The formula has MSE/2 · (1/nᵢ + 1/nⱼ), not MSE · (1/nᵢ + 1/nⱼ). (When nᵢ = nⱼ = n, the simplified form √(MSE/n) absorbs this.)
Students sometimes apply Bonferroni or Tukey's before checking whether the overall ANOVA F-test is significant. These procedures are used after rejecting H₀.
The EWER formula 1 - (1 - α)^m assumes independent tests. The actual EWER for correlated tests (as pairwise comparisons often are) may differ, which is part of why Tukey's uses the Studentized Range distribution rather than relying on this formula directly.
⚠️ The EWER calculation (1 - (1 - α)^m) for a given number of tests is a standard multiple-choice question. For m = 10 and α = 0.05, the answer is approximately 0.4013.
⚠️ Bonferroni's adjusted significance level (α/m) is frequently tested. For 10 comparisons at α = 0.05, the answer is 0.005.
⚠️ Tukey's procedure is a 10-mark long-answer topic. You need to state the steps, show the calculation, build the confidence interval, and draw a conclusion based on whether it contains zero.
⚠️ Know why multiple t-tests are problematic (EWER inflation) and be able to explain it in two to three sentences for short-answer questions.
⚠️ The distinction between what ANOVA tells you (at least two means differ) and what Tukey's tells you (which specific pairs differ) is a common exam theme.
Fill in the blank: The EWER for 10 independent tests at α = 0.05 is approximately ________. 0.4013
True or False: Tukey's procedure should be used before running the ANOVA F-test. False. Tukey's is a post-hoc procedure, used after the F-test rejects H₀.
Fill in the blank: Using the Bonferroni correction for 10 comparisons at α = 0.05, each test is conducted at α = ________. 0.005
True or False: If a Tukey confidence interval for µ₂ - µ₁ contains zero, the two means are significantly different. False. If the interval contains zero, you fail to conclude they are significantly different.
Fill in the blank: The critical value for Tukey's procedure comes from the ________ distribution. Studentized Range (Q) distribution.
Q: Why is performing multiple t-tests to compare pairs of means problematic compared to a single ANOVA?
A: Multiple t-tests inflate the experimentwise error rate (EWER). As the number of comparisons m increases, EWER = 1 - (1 - α)^m grows well beyond the intended α. For example, 10 comparisons at α = 0.05 give EWER ≈ 0.40, meaning a roughly 40% chance of at least one false positive.
Q: When using the Bonferroni correction for 10 comparisons with α = 0.05, what is the adjusted significance level per test?
A: 0.05 / 10 = 0.005.
Q: Outline the steps of Tukey's procedure for comparing two specific group means after a significant ANOVA.
A: (1) Choose the desired EWER (e.g., 0.05). (2) Look up the critical value Q from the Studentized Range table using k and dfE. (3) Compute the confidence interval: (X̄ᵢ - X̄ⱼ) ± Q · √(MSE/2 · (1/nᵢ + 1/nⱼ)). (4) If the interval does not contain zero, conclude the two means are significantly different.
Q: Given k = 4, N = 40, MSE = 12.5, n₁ = n₂ = 10, X̄₁ = 25, X̄₂ = 32, and Q₀.₀₅,₄,₃₆ ≈ 3.81, calculate the Tukey confidence interval for µ₂ - µ₁ and state whether the means differ significantly.
A: Difference = 32 - 25 = 7. Margin of error = 3.81 · √(12.5/2 · (1/10 + 1/10)) = 3.81 · √(1.25) ≈ 3.81 · 1.118 ≈ 4.26. Interval: 7 ± 4.26 = [2.74, 11.26]. Since zero is not in this interval, µ₁ and µ₂ are significantly different.
Q: Which multiple comparison procedure controls the EWER by computing confidence intervals for all pairs of means?
A: Tukey's procedure (Tukey's Honest Significant Difference).
Q: What is the Experimentwise Error Rate, and why does it matter?
A: The EWER is the probability of committing at least one Type I error across a family of tests. It matters because as you run more comparisons, the chance of at least one spurious "significant" result grows rapidly, potentially leading to false conclusions about which groups truly differ.
The EWER concept extends well beyond ANOVA into any setting with multiple tests, including genomics (thousands of gene comparisons) and neuroimaging (thousands of voxel comparisons), where the Bonferroni correction and more advanced methods like the Benjamini-Hochberg procedure are standard. Within STAT 212, Tukey's procedure also connects to the broader idea of simultaneous inference, which reappears in multiple regression when testing several coefficients at once.
Related Terms / Search Tags: multiple comparisons, post-hoc test, Tukey's procedure, Tukey HSD, honest significant difference, Bonferroni correction, experimentwise error rate, EWER, familywise error rate, Type I error, pairwise comparison, Studentized Range distribution, Q critical value, confidence interval for mean difference, alpha adjustment, multiple t-tests problem, STAT 212, Texas A&M, Principles of Statistics II