Source: Comprehensive Overview of Statistical Analysis Techniques (Texas A&M University)
Tags: ANOVA, analysis of variance, one-way ANOVA, F-test, F-statistic, between-group variance, within-group variance, Tukey HSD, post-hoc tests, multiple comparisons, group means, STAT 101
Difficulty: Intermediate | Prerequisites: Familiarity with hypothesis testing (null and alternative hypotheses, p-values), basic understanding of variance and the normal distribution. Multiple linear regression notes from this course are helpful context but not strictly required.
ANOVA sits at the intersection of hypothesis testing and experimental design. Whenever you have a categorical factor (treatment type, group membership, experimental condition) and a continuous outcome, and you want to know whether the groups differ on average, ANOVA is the standard tool. It generalises the two-sample t-test to three or more groups in a single, unified test, which is important because running many pairwise t-tests inflates your false-positive rate. This topic is foundational for understanding more complex designs (two-way ANOVA, repeated measures) and connects tightly to regression, since ANOVA is a special case of linear modelling with categorical predictors.
ANOVA tests whether three or more group means are all equal by comparing how much variation exists between the groups to how much exists within them. If the between-group variation is large relative to the within-group variation, you have evidence that at least one group differs. Post-hoc tests then tell you which specific groups differ.
Analysis of Variance (ANOVA)
A hypothesis-testing framework that compares means across three or more groups by decomposing total variability into between-group and within-group components. Think of it as asking: "Is the spread between group averages bigger than the random noise inside each group?"
One-Way ANOVA
The simplest form of ANOVA, involving a single categorical factor with two or more levels (groups). "One-way" means there is only one grouping variable. In simple terms, you are comparing the means of several groups that differ on one dimension only.
Factor
The categorical independent variable whose effect you are testing (e.g. treatment type, geographic region, teaching method).
Levels (groups)
The distinct categories within a factor. If the factor is "fertiliser type" and you test three brands, the levels are Brand A, Brand B, and Brand C.
Between-group variability
The variation in the data that is attributable to differences among the group means. If the group averages are far apart, this quantity is large.
Within-group variability
The variation that exists among observations within the same group. This reflects individual differences and measurement noise, unrelated to which group an observation belongs to.
F-statistic
The ratio of between-group variability to within-group variability. A large F means the group means are more spread out than you would expect from random variation alone.
F-distribution
The theoretical probability distribution used to determine the critical value for the F-test. The shape depends on two degrees-of-freedom parameters (one for the numerator, one for the denominator).
Null hypothesis (H₀)
In one-way ANOVA, H₀ states that all group population means are equal: μ₁ = μ₂ = ... = μₖ.
Alternative hypothesis (Hₐ)
At least one group mean is different from the others. Note that ANOVA does not tell you which one, only that they are not all the same.
Post-hoc tests
Follow-up tests conducted after a significant ANOVA result to identify which specific pairs of group means differ. They adjust for the fact that you are making multiple comparisons.
Tukey's Honest Significant Difference (Tukey's HSD)
A widely used post-hoc test that compares every pair of group means while controlling the family-wise error rate. In simple terms, it tells you which groups are different from which, without inflating your chance of a false positive.
Family-wise error rate
The probability of making at least one Type I error (false positive) across all the comparisons you perform. ANOVA and post-hoc methods are designed to keep this rate under control.
If you have three groups and run pairwise t-tests (A vs B, A vs C, B vs C), each test carries its own 5% false-positive risk. Across three tests, the combined risk of at least one false positive is notably higher than 5%. With more groups, the problem compounds quickly. ANOVA handles this by conducting a single omnibus test first.
The general model is:
Xᵢⱼ = μᵢ + εᵢⱼ
Xᵢⱼ is the j-th observation in group i.
μᵢ is the true population mean for group i.
εᵢⱼ is the random error for that observation.
The test asks whether all the μᵢ values are equal or whether at least one differs.
The core calculation compares two sources of variation:
F = Between-group variability / Within-group variability
The numerator (Mean Square Between, MSB) measures how much the group means vary around the overall mean.
The denominator (Mean Square Within, MSW) measures how much individual observations vary around their own group mean.
If the groups truly have the same mean, the F-ratio should be close to 1. Values substantially greater than 1 suggest real differences among groups.
Calculate F from your data.
Compare it to the critical value from the F-distribution at your chosen significance level (typically α = 0.05), using the appropriate degrees of freedom.
Equivalently, compute the p-value. If p < α, reject H₀.
Three assumptions underpin the validity of the F-test:
Independence: Each observation is independent of the others. This is primarily a design issue, not something you can fix after the fact.
Normality: The data within each group should be approximately normally distributed. ANOVA is fairly robust to moderate departures from normality, especially with larger sample sizes.
Homogeneity of variances (homoscedasticity): The groups should have roughly equal variances. Levene's test or Bartlett's test can check this. If violated, alternatives such as Welch's ANOVA can be used.
A significant F-test tells you "at least one group differs" but not which one(s). Post-hoc tests fill that gap:
Tukey's HSD is the most common. It compares every pair of means and provides adjusted confidence intervals. It controls the family-wise error rate at your chosen α.
Other options include Bonferroni correction (simple but conservative), Scheffé's method (more flexible but less powerful for pairwise comparisons), and Dunnett's test (when comparing each treatment to a single control group).
Post-hoc tests should only be run after the omnibus ANOVA is significant. Running them on a non-significant ANOVA is poor practice.
One-Way ANOVA model: Xᵢⱼ = μᵢ + εᵢⱼ
F-statistic: F = MSB / MSW
Where:
MSB (Mean Square Between) = SSB / (k - 1), with k = number of groups
MSW (Mean Square Within) = SSW / (N - k), with N = total number of observations
SSB = Σ nᵢ(X̄ᵢ - X̄)², the sum of squared deviations of group means from the grand mean
SSW = ΣΣ (Xᵢⱼ - X̄ᵢ)², the sum of squared deviations of observations from their group means
ANOVA is used in clinical trials to compare the effectiveness of three or more treatments, in agriculture to test whether different fertilisers produce different crop yields, and in education research to compare test scores across teaching methods. Any experiment where you assign subjects to multiple conditions and measure a continuous outcome is a natural fit.
Students often think a significant ANOVA result tells you which group is different. It does not. The F-test only tells you that at least one difference exists. You need a post-hoc test to identify the specific pairs.
Students sometimes believe ANOVA requires equal sample sizes across groups. It does not, though equal (or nearly equal) group sizes improve the robustness of the test to violations of the equal-variance assumption.
Students frequently forget that running multiple t-tests instead of ANOVA inflates the Type I error rate. This is the core motivation for using ANOVA.
Students sometimes assume that failing to reject H₀ means the group means are equal. It means only that you did not find sufficient evidence of a difference, which is not the same thing.
⚠️ Be able to state the null and alternative hypotheses for a one-way ANOVA in both words and symbols.
⚠️ Know how the F-statistic is constructed (between-group variance divided by within-group variance) and what a large vs. small F-value implies.
⚠️ Be able to list and explain the three assumptions (independence, normality, equal variances).
⚠️ Understand why post-hoc tests are necessary after a significant ANOVA and be able to name at least one (Tukey's HSD).
⚠️ Know why running multiple t-tests is problematic and how ANOVA addresses that problem.
True or False: ANOVA can only be used when all groups have the same number of observations.
A: False. ANOVA works with unequal group sizes, though balanced designs are preferred.
Fill in the blank: The F-statistic is the ratio of ________ variability to ________ variability.
A: Between-group; within-group.
True or False: If the ANOVA F-test is not significant, you should still run Tukey's HSD to check for pairwise differences.
A: False. Post-hoc tests should only follow a significant omnibus F-test.
Fill in the blank: The null hypothesis in one-way ANOVA states that all group ________ are equal.
A: Population means.
Q: State the null and alternative hypotheses for a one-way ANOVA comparing the means of four treatment groups.
A: H₀: μ₁ = μ₂ = μ₃ = μ₄ (all group means are equal). Hₐ: At least one group mean differs from the others.
Q: A researcher obtains an F-statistic of 4.82 with a p-value of 0.012. At the α = 0.05 significance level, what is the conclusion?
A: Since p = 0.012 < 0.05, reject H₀. There is sufficient evidence that at least one group mean differs from the others.
Q: Why is Tukey's HSD preferred over running all possible pairwise t-tests after a significant ANOVA?
A: Tukey's HSD controls the family-wise error rate, keeping the overall probability of a false positive at the chosen α level. Multiple t-tests do not make this adjustment, so the true error rate across all comparisons exceeds α.
Q: Name the three assumptions required for a one-way ANOVA and briefly explain each.
A: (1) Independence: observations are not related to one another. (2) Normality: the data within each group are approximately normally distributed. (3) Homogeneity of variances: the population variances of the groups are approximately equal.
Q: An ANOVA compares five groups and produces a significant F-test. How many pairwise comparisons does Tukey's HSD evaluate?
A: 5 choose 2 = 10 pairwise comparisons.
ANOVA is closely related to multiple linear regression. A one-way ANOVA with k groups is mathematically equivalent to a regression model with k - 1 dummy (indicator) variables. Understanding this link helps when you move to two-way ANOVA or ANCOVA, where you combine categorical and continuous predictors in a single model. The assumptions of ANOVA (normality, equal variances, independence) echo the assumptions of OLS regression.
ANOVA, analysis of variance, one-way ANOVA, two-way ANOVA, F-test, F-statistic, F-distribution, between-group variance, within-group variance, MSB, MSW, SSB, SSW, sum of squares, mean square, degrees of freedom, Tukey HSD, post-hoc test, pairwise comparison, multiple comparisons, family-wise error rate, Bonferroni, Scheffé, Dunnett, homogeneity of variances, Levene's test, Welch's ANOVA, group means, null hypothesis, significance level, p-value, STAT 101, principles of statistics