Source: STAT 212, Principles of Statistics II (Texas A&M University)
Tags: Tukey HSD, Tukey's Honestly Significant Difference, post-hoc test, ANOVA, pairwise comparisons, family-wise error rate, confidence intervals, porous carbon, R TukeyHSD, multiple comparisons, STAT 212
Difficulty: Intermediate | Prerequisites: One-way ANOVA basics, confidence intervals, p-values, R fundamentals
Big picture: After running a one-way ANOVA that rejects the null hypothesis, you know that at least one group mean differs from the rest, but you do not know which ones. Tukey's HSD fills that gap. It is one of the most common post-hoc procedures in introductory statistics courses and appears wherever researchers compare three or more treatment groups. You should already be comfortable with one-way ANOVA, the concept of a significance level, and reading basic R output.
Tukey's HSD constructs simultaneous confidence intervals for every pairwise difference in group means, controlling the family-wise error rate at a chosen level (typically 95%). If a confidence interval does not contain zero, that pair of means differs significantly. In the porous carbon experiment, higher sintering temperatures (500 °C, 600 °C) produced significantly different porosity values compared to the baseline (300 °C), while adjacent temperature steps often did not differ.
Tukey's HSD (Honestly Significant Difference)
A post-hoc multiple-comparison procedure used after ANOVA. It calculates a critical range for each pairwise difference in means so that the overall probability of making even one Type I error stays at the chosen confidence level. In simple terms, it is the standard follow-up test that tells you exactly which groups are different from which, while keeping the error rate honest across all comparisons at once.
Family-wise error rate (FWER)
The probability of making at least one Type I error (false positive) across a set of hypothesis tests conducted simultaneously. Think of it as the chance that any one of your multiple comparisons gives you a spurious "significant" result just by luck. Tukey's HSD is designed to hold this rate at your chosen alpha (e.g. 0.05).
Post-hoc test
Any statistical test performed after the initial ANOVA has found a significant overall effect. "Post-hoc" is Latin for "after this." These tests are only appropriate once ANOVA has already rejected the null. In simple terms, ANOVA tells you "something differs." A post-hoc test tells you "here is what differs from what."
Pairwise comparison
A comparison between exactly two group means. With k groups there are k(k-1)/2 unique pairs. For four groups (as in this experiment), that gives six pairs.
Adjusted p-value (p adj)
The p-value for a pairwise comparison after correction for multiple testing. In Tukey's HSD output, if p adj < 0.05, that pair is significantly different at the 95% family-wise confidence level.
Confidence interval containing zero
When a simultaneous confidence interval for a mean difference spans zero (has a negative lower bound and a positive upper bound), you cannot conclude the two means differ. If the entire interval falls on one side of zero, the difference is significant.
Tukey's HSD is applied after a one-way ANOVA rejects the overall null hypothesis (that all group means are equal).
It computes pairwise differences between every pair of group means.
For each pair, it builds a confidence interval around the difference. The width of the interval is based on the studentised range distribution, the within-group variance (MSE), and the sample sizes.
A pair is declared significantly different when its confidence interval does not contain zero, or equivalently, when its adjusted p-value falls below 0.05.
Four sintering temperatures were tested: 300 °C, 400 °C, 500 °C, and 600 °C.
The response variable is a porosity-related measurement ("values") for the resulting porous carbon.
Each temperature group was compared against every other, yielding six pairwise comparisons.
R's TukeyHSD() function returns one row per pair. Each row gives:
diff : the difference in sample means (Group B mean minus Group A mean)
lwr : lower bound of the 95% simultaneous confidence interval
upr : upper bound of the 95% simultaneous confidence interval
p adj : the adjusted p-value for that comparison
Pairs whose confidence interval does NOT contain zero (significant difference):
500 vs 300: diff = -0.77, CI = (-1.42, -0.12), p adj = 0.018
600 vs 300: diff = -1.19, CI = (-1.84, -0.54), p adj = 0.0004
600 vs 400: diff = -1.00, CI = (-1.65, -0.35), p adj = 0.002
Pairs whose confidence interval DOES contain zero (no significant difference):
400 vs 300: diff = -0.19, CI = (-0.84, 0.46), p adj = 0.84
500 vs 400: diff = -0.58, CI = (-1.23, 0.07), p adj = 0.09
600 vs 500: diff = -0.42, CI = (-1.07, 0.23), p adj = 0.29
The pattern: 600 °C differs significantly from both 300 °C and 400 °C. 500 °C differs significantly from 300 °C. Adjacent temperature steps (300 vs 400, 400 vs 500, 500 vs 600) do not reach significance individually.
The plot shows horizontal confidence intervals for each pair, with a dashed vertical line at zero.
If a bar crosses the zero line, that pair is not significantly different.
If a bar sits entirely to one side of zero, the difference is significant.
Wider bars indicate more uncertainty in the estimate.
Number of pairwise comparisons:
C(k, 2) = k! / (2!(k - 2)!)
For k = 4 groups: C(4, 2) = 6
Tukey's HSD critical difference (conceptual form):
HSD = q(alpha, k, df_within) × sqrt(MSE / n)
where q is the studentised range critical value, k is the number of groups, df_within is the within-group degrees of freedom, MSE is the mean square error from ANOVA, and n is the per-group sample size (for balanced designs).
R command:
TukeyHSD(aov(values ~ as.factor(temp)), conf.level = 0.95)
plot(TukeyHSD(aov(values ~ as.factor(temp)), conf.level = 0.95))
Note: as.factor(temp) converts the numeric temperature column into a categorical factor so R treats it as a grouping variable rather than a continuous predictor.
Tukey's HSD is the go-to follow-up whenever researchers compare several treatment conditions, from comparing drug dosages in clinical trials to evaluating manufacturing process settings. In this example, a materials scientist testing sintering temperatures for porous carbon would use the results to identify which temperatures actually produce meaningfully different porosity, rather than assuming every temperature change matters.
Students often believe that a significant ANOVA F-test means every group differs from every other group. It does not. It only tells you that at least one pair differs, which is why you need a post-hoc test.
Students sometimes run pairwise t-tests instead of Tukey's HSD after ANOVA. Multiple uncorrected t-tests inflate the family-wise error rate well beyond 0.05; Tukey's HSD controls for this.
A confidence interval that barely contains zero (e.g. 500 vs 400, with an upper bound of 0.07) is still non-significant at the 95% level. "Close to significant" is not significant.
The sign of the difference (negative values in this output) only reflects the order of subtraction. A diff of -0.77 for "500 - 300" means the 500 °C group has a lower mean, not that the result is somehow negative or bad.
⚠️ You will almost certainly be asked to determine which pairs are significant by checking whether the confidence interval contains zero. Practise reading the table and the plot.
⚠️ Know when to use Tukey's HSD (post-hoc, after a significant ANOVA, all pairwise comparisons needed) versus other procedures like Bonferroni or Dunnett's (which is used when comparing treatments to a single control only).
⚠️ Be able to write and interpret the R commands: aov(), TukeyHSD(), and plot(). Expect questions that show you R output and ask you to state conclusions.
⚠️ The "Contains 0?" column is a common exam shortcut. If you can quickly scan the lower and upper bounds and determine whether they straddle zero, you can answer these questions in seconds.
1. True or False: Tukey's HSD can be used even if the overall ANOVA F-test is not significant.
A: False. Tukey's HSD is a post-hoc test and is only appropriate after ANOVA has rejected the null.
2. Fill in the blank: With 4 groups, Tukey's HSD produces ______ pairwise comparisons.
A: 6
3. True or False: If a Tukey HSD confidence interval for (Group A - Group B) is (-1.65, -0.35), the two groups are NOT significantly different.
A: False. The interval does not contain zero (both bounds are negative), so the groups are significantly different.
4. Fill in the blank: The p adj column in R's TukeyHSD output stands for ______.
A: Adjusted p-value (adjusted for multiple comparisons).
5. True or False: A negative diff value in the Tukey output means the result is invalid.
A: False. It simply means the second group in the pair has a lower mean than the first.
Q: In the porous carbon experiment, which temperature comparisons showed a statistically significant difference at the 95% family-wise confidence level?
A: 500 vs 300, 600 vs 300, and 600 vs 400. These three pairs had adjusted p-values below 0.05 and confidence intervals that did not contain zero.
Q: The confidence interval for the 500 vs 400 comparison is (-1.23, 0.07). What do you conclude, and why?
A: The interval contains zero (the upper bound is positive), so at the 95% family-wise confidence level, there is no significant difference between the mean values at 500 °C and 400 °C.
Q: Why do we use as.factor(temp) in the R code rather than just temp?
A: Temperature is stored as a numeric variable. Without as.factor(), R would treat it as a continuous predictor and fit a regression line. Converting it to a factor tells R to treat each temperature as a separate categorical group, which is what ANOVA requires.
Q: A classmate runs six separate two-sample t-tests instead of Tukey's HSD. What is wrong with this approach?
A: Running multiple uncorrected t-tests inflates the family-wise error rate beyond 0.05. Tukey's HSD uses the studentised range distribution to keep the overall Type I error rate at the stated level across all pairwise comparisons simultaneously.
Q: Looking at the Tukey HSD plot, how do you visually determine whether a pair of means is significantly different?
A: Check whether the horizontal confidence interval bar crosses the vertical dashed line at zero. If it does, the pair is not significantly different. If the entire bar falls to one side of zero, the difference is significant.
This material connects directly to one-way ANOVA (the prerequisite test that precedes Tukey's HSD) and to the broader concept of multiple testing corrections, which reappears in regression diagnostics, genomics, and any setting where many hypotheses are tested at once. The logic of "does the confidence interval contain the null value?" also carries over to two-sample t-tests and proportions tests throughout the course.
Tukey HSD, Tukey's Honestly Significant Difference, post-hoc test, post hoc test, ANOVA follow-up, pairwise comparison, multiple comparison procedure, family-wise error rate, FWER, simultaneous confidence intervals, studentised range, q distribution, adjusted p-value, p adj, porous carbon, sintering temperature, R TukeyHSD function, aov R, STAT 212, Texas A&M, Principles of Statistics II, one-way ANOVA post-hoc, which groups differ