Difficulty: Intermediate to Advanced | Prerequisites: Foundations doc and One-Sample Tests doc.
Big Picture
Two-sample tests extend the one-sample framework to compare two populations. The central question becomes: is the difference between two means (or two variances) statistically significant? The choice of test depends on whether variances are known, equal, or unequal. This doc also covers the chi-squared independence test for categorical data, and three important theoretical tools: the Neyman-Pearson lemma (optimal testing), the Bonferroni correction (multiple comparisons), and the Kolmogorov-Smirnov test (distribution fitting).
TL;DR
Comparing two means with known variances: Z-test. Unknown but equal variances: pooled t-test. Unknown and unequal variances: Welch's t-test. Comparing two variances: F-test. Testing independence in a contingency table: χ²-test. The Neyman-Pearson lemma tells you the most powerful test possible; Bonferroni corrects for testing many hypotheses at once; Kolmogorov-Smirnov checks whether data follow a specified distribution.
Pooled variance (Sp²)
A weighted average of the two sample variances, used when the two populations are assumed to have equal variance: Sp² = [(n₁−1)S₁² + (n₂−1)S₂²] / (n₁ + n₂ − 2).
Think of it as: a single "best guess" variance that combines information from both samples.
Welch's approximation (ν)
The degrees of freedom for the two-sample t-test when variances are unequal. Computed via Satterthwaite's formula, which is messier than the pooled case but avoids the equal-variance assumption.
F-statistic (two-sample)
F = S₁² / S₂² ~ F_{n₁−1, n₂−1} under H₀: σ₁² = σ₂². Tests whether two populations have the same variance.
χ²-test for independence
Tests whether two categorical variables in a contingency table are independent. The test statistic compares observed cell counts to expected counts under independence.
Neyman-Pearson lemma
For simple H₀ vs simple H_A, the most powerful test at level α is the likelihood ratio test: reject H₀ when L(θ_A)/L(θ₀) > k, for some threshold k.
In simple terms, the best possible test compares how likely the data are under each hypothesis.
Bonferroni correction
When testing m hypotheses simultaneously, reject the ith hypothesis only if pᵢ < α/m. Controls the family-wise error rate.
Kolmogorov-Smirnov (KS) test
Measures the supremum of the absolute difference between the empirical CDF and a reference CDF: D = sup |Fₙ(x) − F(x)|. Reject if D exceeds a critical value.
Think of it as: how far does the data's distribution stray from the theoretical one at any point?
When to use: Comparing two population means when both populations are normal with unknown but assumed equal variances.
Setup: H₀: μ₁ − μ₂ = Δ₀ (often Δ₀ = 0)
Test statistic:
T₀ = (X̄₁ − X̄₂ − Δ₀) / (Sp · √(1/n₁ + 1/n₂)) ~ t_{n₁+n₂−2} under H₀
where Sp² = [(n₁−1)S₁² + (n₂−1)S₂²] / (n₁ + n₂ − 2)
and the pooled standard deviation is also expressible as:
U = [(m+n−2)^{1/2} · (X̄ₘ − Ȳₙ)] / [(1/m + 1/n)^{1/2} · (S₁² + S₂²)^{1/2}]
where S₁² = Σ(Xᵢ − X̄ₘ)² and S₂² = Σ(Yᵢ − Ȳₙ)².
Rejection rules:
H_A: μ₁ − μ₂ ≠ Δ₀: reject if |T₀| > t_{n₁+n₂−2}(1 − α/2)
H_A: μ₁ − μ₂ > Δ₀: reject if T₀ > t_{n₁+n₂−2}(1 − α)
H_A: μ₁ − μ₂ < Δ₀: reject if T₀ < t_{n₁+n₂−2}(α)
Worked example from the notes
Given: X̄ = 18.18, S₁² = 12.61, Ȳ = 17.32, S₂² = 11.01, n₁ = n₂ = 5 (inferred), testing H₀: Δ₀ = 0 at α = 0.05.
U₀ = (5+5−2)^{1/2} · (18.18 − 17.32) / [(1/5 + 1/5)^{1/2} · (12.61 + 11.01)^{1/2}] = 0.791
Degrees of freedom: n₁ + n₂ − 2 = 8. Critical value: t₈(0.975) = 2.306.
Since |0.791| < 2.306, the null is not rejected. The two means are not significantly different at α = 0.05.
When to use: Comparing two population means when the variances are unknown and you cannot assume they are equal.
Test statistic:
W = (X̄₁ − X̄₂ − Δ₀) / √(S₁²/n₁ + S₂²/n₂) ~ tᵥ under H₀
Degrees of freedom (Satterthwaite approximation):
ν = (S₁²/n₁ + S₂²/n₂)² / [(S₁²/n₁)²/(n₁−1) + (S₂²/n₂)²/(n₂−1)]
Round ν down to the nearest integer for table lookups.
Rejection rules: Same structure as the pooled test, but using tᵥ instead of t_{n₁+n₂−2}.
When to choose Welch's over pooled:
If you have reason to believe the two populations have different variances (or if an F-test rejects equality of variances), use Welch's. In practice, Welch's is often the safer default because it does not assume equal variances and still works well when variances happen to be equal.
When to use: Comparing two means when both population variances σ₁² and σ₂² are known.
Test statistic:
Z₀ = (X̄₁ − X̄₂ − Δ₀) / √(σ₁²/n₁ + σ₂²/n₂) ~ N(0, 1) under H₀
P-values:
H_A: μ₁ − μ₂ ≠ Δ₀: 2[1 − Φ(|Z₀|)]
H_A: μ₁ − μ₂ > Δ₀: 1 − Φ(Z₀)
H_A: μ₁ − μ₂ < Δ₀: Φ(Z₀)
Or equivalently, using the Φ⁻¹ quantiles: reject if the observed Z exceeds the critical value from the same direction as the alternative.
When to use: Testing whether two normal populations have the same variance.
Test statistic:
F = S₁² / S₂² ~ F_{n₁−1, n₂−1} under H₀: σ₁² = σ₂²
Rejection rules:
H_A: σ₁² ≠ σ₂²: reject if F > F⁻¹(1 − α/2) or F < F⁻¹(α/2). P-value = 2 · min{F(f₀), 1 − F(f₀)}
H_A: σ₁² > σ₂²: reject if F > F⁻¹(1 − α). P-value = 1 − F(f₀)
H_A: σ₁² < σ₂²: reject if F < F⁻¹(α). P-value = F(f₀)
Real-world use: The F-test is often a preliminary step. Before running a pooled t-test, you check whether the equal-variance assumption holds. If the F-test rejects, switch to Welch's t-test.
When to use: You have a contingency table of counts for two categorical variables and want to test whether the variables are independent.
Setup:
H₀: Pᵢⱼ = Pᵢ · Pⱼ (row and column variables are independent)
H_A: the null is not true (some association exists)
Expected counts under the null:
Êᵢⱼ = (Nᵢ₊ · N₊ⱼ) / n
where Nᵢ₊ is the ith row total, N₊ⱼ is the jth column total, and n is the grand total.
Test statistic:
Q = Σᵢ Σⱼ (Nᵢⱼ − Êᵢⱼ)² / Êᵢⱼ ~ χ²_{(R−1)(C−1)} under H₀
where R is the number of rows and C the number of columns.
Degrees of freedom: (R − 1)(C − 1)
Rejection rule: Reject H₀ if Q > χ²_{(R−1)(C−1)}(1 − α). This is always a right-tailed test because larger Q means the observed counts deviate more from what independence predicts.
Real-world application: This test is how you determine whether two categorical variables are associated. For example, testing whether smoking status is independent of lung disease status in a medical study, or whether customer segment is independent of purchase category.
For testing simple H₀: θ = θ₀ against simple H_A: θ = θ₁, the uniformly most powerful (UMP) test at level α is the likelihood ratio test:
Reject H₀ if L(θ₁) / L(θ₀) > k, where k is chosen so that P(reject | H₀) = α.
Equivalently, the UMP test has the form: "reject if T(x) < k" for a sufficient statistic T.
This lemma is the theoretical justification for why likelihood-based tests are optimal. It does not extend directly to composite hypotheses (where the alternative is a range of values), which is why we need other tools.
When testing m hypotheses simultaneously at family-wise error rate α, reject the ith null hypothesis only if its p-value satisfies pᵢ < α/m.
This is conservative (it controls the probability of making even one false rejection across all m tests). The trade-off is reduced power: as m grows, the threshold for each individual test becomes very stringent.
Tests whether a sample comes from a specified continuous distribution.
Test statistic:
D = sup_x |Fₙ(x) − F(x)|
where Fₙ is the empirical CDF and F is the hypothesised CDF.
Reject if D exceeds the critical value from KS tables (which depends on n and α).
The KS test is non-parametric: it makes no assumption about the form of the distribution being tested. It is particularly useful when you want to check whether data follow a specific theoretical distribution (normal, exponential, uniform, etc.) without committing to a parametric test.
A confidence interval at level 1 − α satisfies:
P(A(X₁, ..., Xₙ) ≤ θ₀ ≤ B(X₁, ..., Xₙ)) ≥ 1 − α
For each test in this course, the confidence interval is constructed by inverting the test: collecting all parameter values that would not be rejected at level α.
Misconceptions:
Students often use the pooled t-test by default without checking whether the equal-variance assumption holds. Always consider running an F-test first, or default to Welch's.
The F-distribution is not symmetric. The two-sided F-test requires looking up both tails separately.
In the χ²-test for independence, small expected cell counts (below 5) make the approximation unreliable. This is a common exam trap.
The Bonferroni correction controls the family-wise error rate, not the false discovery rate. These are different things and students often conflate them.
Exam flags:
⚠️ Know how to compute the pooled variance and set up a two-sample t-test from raw summary statistics.
⚠️ Be able to state and apply the Neyman-Pearson lemma for a simple-vs-simple test.
⚠️ Expect a contingency table problem requiring you to compute expected counts and the Q statistic.
⚠️ The Bonferroni threshold α/m is a favourite short-answer question.
⚠️ Know the KS test statistic formula and what "sup" means in this context (the largest vertical gap between the empirical and theoretical CDFs).
True or False: The pooled t-test assumes equal population variances.
True.
Fill in the blank: The degrees of freedom for the pooled two-sample t-test are ______.
n₁ + n₂ − 2
True or False: The F-test for comparing variances is always one-tailed.
False. It can be one-tailed or two-tailed depending on the alternative.
Fill in the blank: In the χ²-test for independence, degrees of freedom = ______.
(R − 1)(C − 1)
True or False: The Neyman-Pearson lemma guarantees the most powerful test for composite alternatives.
False. It applies to simple H₀ vs simple H_A only.
Q: Two independent samples from normal populations give X̄₁ = 25, S₁² = 9, n₁ = 10 and X̄₂ = 22, S₂² = 11, n₂ = 12. Assuming equal variances, test H₀: μ₁ − μ₂ = 0 at α = 0.05.
A: Sp² = [(9)(9) + (11)(11)] / (10 + 12 − 2) = (81 + 121) / 20 = 10.1. T₀ = (25 − 22) / √(10.1 · (1/10 + 1/12)) = 3 / √(10.1 · 0.1833) = 3 / √1.851 = 3 / 1.361 = 2.204. With df = 20, t₂₀(0.975) ≈ 2.086. Since 2.204 > 2.086, reject H₀.
Q: When should you use Welch's t-test instead of the pooled t-test?
A: When the two population variances are unknown and you cannot assume they are equal. In practice, if an F-test rejects the null of equal variances, or if the sample variances are very different, use Welch's.
Q: A 3×2 contingency table has expected counts all above 5. What are the degrees of freedom for the χ²-test?
A: (3 − 1)(2 − 1) = 2.
Q: State the Neyman-Pearson lemma in one sentence.
A: For simple H₀ vs simple H_A, the most powerful test at level α rejects H₀ when the likelihood ratio L(θ_A)/L(θ₀) exceeds a threshold chosen to give size α.
Q: You are testing 20 hypotheses simultaneously with a family-wise error rate of 0.05. What is the Bonferroni-adjusted significance threshold for each individual test?
A: α/m = 0.05/20 = 0.0025. Reject an individual hypothesis only if its p-value is below 0.0025.
Two-sample t-test, pooled t-test, Welch's t-test, Satterthwaite degrees of freedom, two-sample Z-test, F-test, ratio of variances, chi-squared test for independence, contingency table, expected counts, Neyman-Pearson lemma, likelihood ratio test, uniformly most powerful test, UMP, Bonferroni correction, family-wise error rate, multiple comparisons, Kolmogorov-Smirnov test, KS test, empirical CDF, supremum, confidence interval, Analytical Statistics NYU