Difficulty: Intermediate | Prerequisites: One-sample hypothesis testing (Part 1 of these notes), familiarity with t and chi-squared distributions
Two-sample tests extend everything you learnt in one-sample testing to comparisons between two populations. Instead of asking "does the population mean equal some value?" you are now asking "do these two populations have the same mean, or the same variance?"
This is where hypothesis testing becomes practical. Most real questions in science, engineering, and business involve comparing two groups: a treatment vs. a control, this year vs. last year, machine A vs. machine B. The logic (set up hypotheses, compute a test statistic, compare to a distribution) is identical to the one-sample case; only the formulas change.
You should already be comfortable with the Z-test, t-test, and chi-squared test from Part 1 before tackling this material.
Two-sample tests let you compare means or variances between two independent groups. When both population variances are known, use a Z-test on the difference in means. When variances are unknown, use a pooled t-test (if you can assume equal variances) or the Welch t-test (if you cannot). To test whether two populations have the same variance, use the F-test.
Pooled variance (Sp²)
A weighted average of the two sample variances, used when you assume both populations have equal variance. The formula is Sp² = [(n₁ − 1)S₁² + (n₂ − 1)S₂²] / (n₁ + n₂ − 2). Think of it as combining the information from both samples into a single, better estimate of the common variance.
Welch's t-test / Welch approximation
A modification of the two-sample t-test that does not assume equal variances. It uses the individual sample variances S₁²/n₁ and S₂²/n₂ directly and adjusts the degrees of freedom downward. In simple terms, it is the safer default when you are not sure both groups have the same spread.
Satterthwaite degrees of freedom (ν)
The adjusted degrees of freedom used in Welch's t-test. The formula is ν = (S₁²/n₁ + S₂²/n₂)² / [(S₁²/n₁)²/(n₁−1) + (S₂²/n₂)²/(n₂−1)]. It is typically not an integer, so you round down or use software. Think of it as the price you pay for not assuming equal variances: fewer effective degrees of freedom.
F-distribution
The distribution of the ratio of two independent chi-squared random variables, each divided by its degrees of freedom. It arises naturally when comparing two sample variances. In simple terms, if you divide one sample variance by another, the result follows an F-distribution under the null that the population variances are equal.
Δ₀ (delta-naught)
The hypothesised difference in means under H₀. Most often Δ₀ = 0 (testing whether the two means are equal), but the framework allows any value.
When to use: You have two independent samples from normal populations and both population variances σ₁² and σ₂² are known. You want to test whether the difference in population means equals some hypothesised value Δ₀.
Hypotheses
Two-sided: H₀: μ₁ − μ₂ = Δ₀ vs. H₁: μ₁ − μ₂ ≠ Δ₀
Right-sided: H₁: μ₁ − μ₂ > Δ₀
Left-sided: H₁: μ₁ − μ₂ < Δ₀
Test statistic
Z₀ = (X̄₁ − X̄₂ − Δ₀) / √(σ₁²/n₁ + σ₂²/n₂), which follows N(0,1) under H₀.
Rejection rules and p-values
Identical in structure to the one-sample Z-test: compare z₀ to the standard normal quantiles and compute p-values accordingly.
When to use: Both population variances are unknown, but you have reason to believe they are equal (σ₁² = σ₂²). This is sometimes called the "equal-variance t-test."
Pooled variance estimate
Sp² = [(n₁ − 1)S₁² + (n₂ − 1)S₂²] / (n₁ + n₂ − 2)
This combines both samples' information. The degrees of freedom for the pooled estimate are n₁ + n₂ − 2.
Test statistic
T₀ = (X̄₁ − X̄₂ − Δ₀) / [Sp × √(1/n₁ + 1/n₂)], which follows t(n₁ + n₂ − 2) under H₀.
Rejection rules and p-values
Same structure as the one-sample t-test, but with n₁ + n₂ − 2 degrees of freedom.
When is the equal-variance assumption reasonable? If the ratio of the larger sample variance to the smaller is less than about 2, the pooled test tends to perform well. For a formal check, run the F-test on the ratio of variances (covered below).
When to use: Both population variances are unknown and you cannot assume they are equal. This is the safer choice when in doubt.
Test statistic
T₀ = (X̄₁ − X̄₂ − Δ₀) / √(S₁²/n₁ + S₂²/n₂), which follows approximately t(ν) under H₀.
The denominator uses each sample's own variance estimate rather than a pooled one.
Satterthwaite degrees of freedom
ν = (S₁²/n₁ + S₂²/n₂)² / [(S₁²/n₁)²/(n₁ − 1) + (S₂²/n₂)²/(n₂ − 1)]
This is generally not an integer. Round down to be conservative, or use software.
Rejection rules and p-values
Same structure as any t-test, but using ν (Satterthwaite) degrees of freedom instead of a clean n₁ + n₂ − 2.
Practical note: Many software packages default to Welch's test because it is valid whether or not the variances are equal. The pooled test is slightly more powerful when the equal-variance assumption holds, but it can be misleading when it does not.
When to use: You have two independent samples from normal populations and want to test whether the population variances are equal. This is also used as a preliminary check before deciding between the pooled and Welch t-tests.
Hypotheses
Two-sided: H₀: σ₁² = σ₂² vs. H₁: σ₁² ≠ σ₂²
Right-sided: H₁: σ₁² > σ₂²
Left-sided: H₁: σ₁² < σ₂²
Test statistic
F₀ = S₁² / S₂², which follows F(n₁ − 1, n₂ − 1) under H₀.
The F-distribution has two parameters: the numerator degrees of freedom (n₁ − 1) and the denominator degrees of freedom (n₂ − 1). It is not symmetric.
Rejection rules and p-values
Two-sided: reject if F₀ > F(1 − α/2) or F₀ < F(α/2). p-value = 2 × min[F_cdf(f₀), 1 − F_cdf(f₀)].
Right-sided: reject if F₀ > F(1 − α). p-value = 1 − F_cdf(f₀).
Left-sided: reject if F₀ < F(α). p-value = F_cdf(f₀).
Important: Like the chi-squared variance test, the F-test is sensitive to departures from normality. It should be used with caution if the populations are not close to normal.
Test | Test Statistic | Null Distribution | Degrees of Freedom |
|---|---|---|---|
Z-test (σ₁², σ₂² known) | Z₀ = (X̄₁ − X̄₂ − Δ₀) / √(σ₁²/n₁ + σ₂²/n₂) | N(0,1) | n/a |
Pooled t-test (equal var.) | T₀ = (X̄₁ − X̄₂ − Δ₀) / [Sp√(1/n₁ + 1/n₂)] | t(n₁+n₂−2) | n₁ + n₂ − 2 |
Welch t-test (unequal var.) | T₀ = (X̄₁ − X̄₂ − Δ₀) / √(S₁²/n₁ + S₂²/n₂) | t(ν) | Satterthwaite ν |
F-test (variance ratio) | F₀ = S₁² / S₂² | F(n₁−1, n₂−1) | (n₁−1, n₂−1) |
How chi-squared, t, and F distributions relate
If X₁, ..., Xₙ are i.i.d. N(0,1), then X₁² + ... + Xₙ² ~ χ²(n).
If X ~ N(0,1) and Y ~ χ²(n) are independent, then X / √(Y/n) ~ t(n).
If X ~ χ²(n) and Y ~ χ²(m) are independent, then (X/n) / (Y/m) ~ F(n,m).
The t-distribution is built from a normal divided by a scaled chi-squared. The F-distribution is built from a ratio of two chi-squareds. These relationships explain why the t-test and F-test have the forms they do.
Students often assume the pooled t-test is always correct. It is only appropriate when the equal-variance assumption holds. If in doubt, use Welch's test or run the F-test first.
Students sometimes swap the numerator and denominator in the F-test and then wonder why their F₀ is less than 1. Convention places the larger sample variance in the numerator for a right-sided test, but for the two-sided test the formula is simply S₁²/S₂² as labelled.
The Satterthwaite degrees of freedom can look intimidating, but it is just arithmetic. Students sometimes skip it and use n₁ + n₂ − 2 for the Welch test, which is incorrect.
Students sometimes forget that two-sample tests assume independence between the two samples. If the same subjects appear in both groups (before/after measurements, for instance), you need a paired test, not a two-sample test.
⚠️ Be able to identify which two-sample test to use from the problem statement. "Variances are known" means Z-test. "Assume equal variances" means pooled t-test. "Do not assume equal variances" means Welch.
⚠️ Know how to compute the pooled variance Sp². This is a very common calculation question.
⚠️ You may be asked to compute Satterthwaite degrees of freedom by hand. Practise the formula so you do not waste time on exam day.
⚠️ The F-test is often tested as a preliminary step: "First test whether the variances are equal, then choose the appropriate t-test." Be ready for a two-part problem.
⚠️ Pay attention to the order of the F-distribution parameters. F(n₁ − 1, n₂ − 1) is not the same as F(n₂ − 1, n₁ − 1).
True or false: The pooled t-test and the Welch t-test always give the same result.
False. They give similar results when variances are roughly equal, but can differ substantially when variances are unequal.
Fill in the blank: The pooled variance Sp² uses ___ degrees of freedom.
n₁ + n₂ − 2.
True or false: The F-distribution is symmetric.
False. The F-distribution is right-skewed.
Fill in the blank: In the F-test for two variances, the test statistic is F₀ = S₁² / ___ .
S₂².
True or false: If the two samples are paired (same subjects measured twice), you should use a two-sample t-test.
False. Paired data requires a paired t-test, which analyses the differences within each pair as a single sample.
Q: Two machines produce bolts. A sample of 40 bolts from Machine 1 has X̄₁ = 10.2 mm with σ₁ = 0.5 mm. A sample of 50 bolts from Machine 2 has X̄₂ = 10.0 mm with σ₂ = 0.6 mm. At α = 0.05, is there a significant difference in mean bolt diameter?
A: Z₀ = (10.2 − 10.0 − 0) / √(0.25/40 + 0.36/50) = 0.2 / √(0.00625 + 0.0072) = 0.2 / √0.01345 = 0.2 / 0.116 ≈ 1.72. The critical value for a two-sided test at α = 0.05 is 1.96. Since 1.72 < 1.96, fail to reject H₀. There is insufficient evidence of a difference in mean diameters.
Q: Two groups are tested. Group 1 (n₁ = 12) has S₁² = 24. Group 2 (n₂ = 15) has S₂² = 10. Test H₀: σ₁² = σ₂² vs. H₁: σ₁² ≠ σ₂² at α = 0.05.
A: F₀ = 24/10 = 2.4, with df = (11, 14). Look up F(11, 14) critical values at α/2 = 0.025 in each tail. If the upper critical value exceeds 2.4, fail to reject. (For this pair of degrees of freedom, the upper critical value is approximately 3.09, so 2.4 < 3.09 and we fail to reject H₀.)
Q: When should you use the Welch t-test instead of the pooled t-test?
A: Use the Welch t-test when the population variances are unknown and you cannot assume they are equal. If the F-test rejects equal variances, or if the sample variances look substantially different, the Welch test is the safer choice.
Q: Explain what the Satterthwaite degrees of freedom represent and why they matter.
A: Satterthwaite degrees of freedom adjust for the fact that the Welch test statistic does not follow an exact t-distribution. The formula produces a value (often non-integer) that is smaller than the n₁ + n₂ − 2 used in the pooled test, making rejection harder. This reflects the additional uncertainty introduced by estimating two separate variances.
The F-test here is a building block for analysis of variance (ANOVA), which generalises the two-sample comparison to three or more groups. ANOVA uses F-statistics extensively.
The pooled variance concept reappears in linear regression, where the residual variance is essentially a pooled estimate across all observations.
If the two samples are not independent (e.g. before/after measurements on the same subjects), these tests do not apply. You would instead use a paired t-test, which reduces the problem to a one-sample test on the differences.
two-sample test, difference in means, pooled t-test, equal variance t-test, Welch t-test, unequal variance t-test, Satterthwaite degrees of freedom, pooled variance, F-test, ratio of variances, F-distribution, chi-squared distribution, t-distribution, independent samples, paired test, ANOVA, analysis of variance, two-tailed test, one-tailed test, critical value, p-value, significance level, NYU analytical statistics, statistical inference, comparing two populations