Source: Principles of Statistics I, Texas A&M / Tamhane & Dunlop Ch. 5, 6.1, 14.6, 15.1
Tags: sample standard deviation, sample variance, chi-square distribution, sample quantile, sample proportion, binomial distribution, normal approximation, continuity correction, hypergeometric, sample minimum, sample maximum, extreme value distribution, simulation study
The sample variance S² is unbiased for σ², but S itself is biased low (the bias shrinks with n). The sampling distribution of S connects to the chi-square distribution for normal data. Sample quantiles and the sample proportion both have known asymptotic distributions via the CLT. The normal approximation to the binomial requires min(np, n(1−p)) ≥ 5, and a continuity correction of ±0.5 improves accuracy in the middle of the distribution. Simulation studies confirm that asymptotic approximations improve with n and work best when the population is close to symmetric.
Sample variance (S²)
S² = (1/(n−1)) Σ(Xᵢ − X-bar)². Unbiased for σ² for all n.
Sample standard deviation (S)
S = √S². A biased estimator of σ: it systematically underestimates σ, though the bias decreases as n grows.
Chi-square distribution
If X₁, …, Xₙ are iid N(μ, σ²), then (n−1)S²/σ² follows a chi-square distribution with df = n − 1. This is the foundational result for inference about variance from normal data.
Sample quantile Q-hat(u)
An estimator of the population u-th quantile Q(u). For values of u not too close to 0 or 1, its asymptotic distribution is normal.
Sample proportion (p-hat)
p-hat = Y/n, where Y is the count of "successes" in n trials. Unbiased for p under both sampling with and without replacement.
Continuity correction
An adjustment of ±0.5 applied when using a continuous (normal) distribution to approximate a discrete (binomial) distribution. It improves accuracy in the centre of the distribution but can worsen it in the tails.
Hypergeometric distribution
The distribution of the count of successes when sampling without replacement from a finite population. Replaces the binomial in finite-population settings.
Sample minimum and maximum
The smallest and largest order statistics, m_n = X₍₁₎ and M_n = X₍ₙ₎. Their distributions follow from powers of the CDF and survival function.
Key properties
E[S²] = σ² for all n (S² is unbiased)
E[S] ≠ σ for any finite n (S is biased low)
Var[S] depends on the population distribution F
Under normality: exact results
If the population is N(μ, σ²):
(n−1)S²/σ² ~ χ²(n−1)
E[S] = cₙσ, where cₙ = √(2/(n−1)) × Γ(n/2) / Γ((n−1)/2)
Var[S] = σ²(1 − cₙ²)
Both cₙ → 1 and Var[S] → 0 as n → ∞
Bias of S as an estimator of σ (normal data)
Bias = (cₙ − 1)σ, which is always negative:
n | Bias |
|---|---|
2 | −0.202σ |
3 | −0.114σ |
5 | −0.060σ |
10 | −0.027σ |
25 | −0.0104σ |
100 | −0.00252σ |
250 | −0.00100σ |
The bias is negligible for n ≥ 25 or so.
Asymptotic distribution (large n)
S is approximately N(σ, (μ₄ − σ⁴)/(4nσ²))
Asymptotic mean: μ_A = σ
Asymptotic standard deviation: σ_A = √(μ₄ − σ⁴) / (2σ√n)
Here μ₄ is the fourth central moment of the population.
Effect of correlation on S²
For stationary data with covariance function R(k):
E[S²] = σ² = R(0) (still unbiased)
Var(S²) = (2/n) Σ (1 − i/n) R²(i)
Correlation affects the precision of S² but not its expected value.
Smoothed definition
For 1/(2n) ≤ u ≤ 1 − 1/(2n), with nu + 0.5 = k + r where k is an integer and 0 < r < 1:
Q-hat(u) = Y₍ₖ₎ + r[Y₍ₖ₊₁₎ − Y₍ₖ₎]
Exact moments
The moments depend on the population CDF through the distribution of order statistics:
f₍ₖ₎(y) = n! / [(k−1)!(n−k)!] × [F(y)]^(k−1) × [1−F(y)]^(n−k) × f(y)
Asymptotic distribution (large n)
Q-hat(u) is approximately N(Q(u), u(1−u) / [n × (f(Q(u)))²])
Asymptotic standard deviation: σ_A = √(u(1−u)) / [f(Q(u)) × √n]
For the median (u = 0.5) from a N(μ, σ²) distribution, this simplifies to:
σ_A = σ√π / √(2n)
In practice, you need to estimate f(Q(u)) to compute the asymptotic standard error, which adds an extra layer of estimation.
CDFs and PDFs
For iid X₁, …, Xₙ with CDF F:
Maximum M_n: G_n(y) = [F(y)]ⁿ, with pdf g_n(y) = n f(y)[F(y)]^(n−1)
Minimum m_n: H_n(y) = 1 − [1−F(y)]ⁿ, with pdf h_n(y) = n f(y)[1−F(y)]^(n−1)
Bounds on expectations
If the population has mean μ and finite variance σ²:
E(M_n) ≤ μ + (n−1)σ / √(2n−1)
E(m_n) ≥ μ − (n−1)σ / √(2n−1)
Asymptotic results (extreme value theory)
The standardised maximum (M_n − a_n)/b_n converges to one of three extreme value distributions:
Frechet (G₁): for heavy-tailed distributions
Weibull-type (G₂): for distributions with a finite upper endpoint
Gumbel (G₃): G₃(y) = exp(−e^(−y)), for distributions like normal, exponential, and Weibull (after log transform)
The asymptotic distribution for the minimum is obtained by using Xᵢ = −Yᵢ.
Setup
Population units are one of two types (A or B). Let p = proportion of Type A. p-hat = Y/n, where Y = number of Type A in the sample.
Exact distributions
Sampling with replacement: np-hat ~ Binomial(n, p)
Sampling without replacement: np-hat ~ Hypergeometric
Mean and variance
E[p-hat] = p (unbiased in both cases)
With replacement: Var[p-hat] = p(1−p)/n
Without replacement: Var[p-hat] = ((N−n)/(N−1)) × p(1−p)/n
When n/N < 0.05, the fpcf ≈ 1, so Var[p-hat] ≈ p(1−p)/n regardless
Normal approximation (large n)
p-hat is approximately N(p, p(1−p)/n)
The approximation should only be used when min(np, n(1−p)) ≥ 5.
Continuity correction
Because we are approximating a discrete distribution with a continuous one:
P[p-hat = y] = P[np-hat = ny] ≈ P[ny − 0.5 ≤ Y ≤ ny + 0.5]
The general formula with continuity correction:
P[y₁ ≤ p-hat ≤ y₂] ≈ Φ((ny₂ + 0.5 − np)/√(np(1−p))) − Φ((ny₁ − 0.5 − np)/√(np(1−p)))
Example: Y ~ Binomial(100, 0.2), find P(Y ≤ 25)
Exact: P(Y ≤ 25) = 0.9125
Normal without correction: 0.8944
Normal with correction: 0.9154
The continuity correction is more accurate in the middle of the distribution but can be less accurate in the tails.
Simulations of 10,000 samples at sizes n = 10, 25, 100 from three distributions compared asymptotic predictions with observed behaviour.
N(0,1): symmetric, light tails
Asymptotic approximations are excellent even at small n. The number of outliers in boxplots of 10,000 values is close to the expected 70 for a normal distribution.
Weibull(0.61, 0.62): highly right-skewed (skewness = 4.38)
Asymptotic approximations for the mean are reasonable, but median and standard deviation approximations are poor at small n.
The simulation means for the median and S are substantially different from asymptotic predictions at n = 10.
Boxplots show extreme right skew with many upper outliers, even at n = 100.
t(5): symmetric, heavy tails
Mean and median approximations are reasonable.
The standard deviation shows many outliers (239+ per 10,000 samples) because the t(5) has heavy tails, so extreme values frequently inflate S.
The ARE comparison is visible: median and mean have similar variability (ARE ≈ 0.96 for df = 5).
The overarching lesson is that the normal approximation improves with n and works best when the population is close to symmetric with moderate tails.
Chi-square result: (n−1)S²/σ² ~ χ²(n−1) when sampling from N(μ, σ²)
Asymptotic StDev of S: σ_A = √(μ₄ − σ⁴) / (2σ√n)
Asymptotic StDev of Q-hat(u): σ_A = √(u(1−u)) / [f(Q(u))√n]
Variance of p-hat: Var(p-hat) = p(1−p)/n
Min(np, n(1−p)) ≥ 5 rule for normal approximation to binomial
CDF of sample max: G_n(y) = [F(y)]ⁿ
CDF of sample min: H_n(y) = 1 − [1 − F(y)]ⁿ
⚠️ S² is unbiased for σ², but S is biased low. Know why: Jensen's inequality (the square root is a concave function).
⚠️ The chi-square result for (n−1)S²/σ² requires normality. Do not apply it to non-normal populations.
⚠️ The rule min(np, n(1−p)) ≥ 5 is the standard condition for using the normal approximation to the binomial. Be ready to check it.
⚠️ The continuity correction adds 0.5 to the numerator for upper bounds and subtracts 0.5 for lower bounds. It helps in the middle of the distribution but can hurt in the tails.
⚠️ The asymptotic variance of sample quantiles depends on the density at the quantile. If the density is low at that point (e.g. in the tails), the estimator is imprecise.
⚠️ Simulation results show that skewed populations (like Weibull) need much larger n before the CLT approximation becomes reliable.
Q: Is S² an unbiased estimator of σ²? Is S an unbiased estimator of σ?
A: S² is unbiased for σ² for all n. S is biased: it underestimates σ, though the bias decreases with n.
Q: If X₁, …, X₂₀ are iid N(μ, σ²), what distribution does 19S²/σ² follow?
A: Chi-square with 19 degrees of freedom.
Q: State the condition for using the normal approximation to the binomial.
A: min(np, n(1−p)) ≥ 5.
Q: For a Binomial(100, 0.2) distribution, compute the normal approximation to P(Y ≤ 25) with the continuity correction.
A: P(Y ≤ 25) ≈ Φ((25 + 0.5 − 20)/4) = Φ(5.5/4) = Φ(1.375) ≈ 0.915.
Q: What is the CDF of the sample maximum for n iid observations with CDF F?
A: G_n(y) = [F(y)]ⁿ.
Q: Why does the normal approximation work poorly for the sample median at small n from a Weibull distribution?
A: The Weibull(0.61, 0.62) is highly right-skewed (skewness = 4.38). The sample median's distribution retains much of this skewness at small n, and the CLT has not yet "kicked in" to produce approximate normality.
sample standard deviation, sample variance, chi-square, unbiased estimator, bias of S, fourth central moment, sample quantile, order statistic, sample proportion, binomial distribution, hypergeometric, normal approximation, continuity correction, sample minimum, sample maximum, extreme value distribution, Frechet, Gumbel, Weibull, simulation study, N(0 1), t distribution, skewness, outliers, boxplot, asymptotic standard deviation, Tamhane Dunlop, STAT Texas A&M, Handout 10