Source: Principles of Statistics I, Texas A&M / Tamhane & Dunlop Ch. 5, 6.1
Tags: sampling distribution, estimator, bias, unbiased estimator, MSE, mean squared error, variance of statistic, expected value of statistic, enumeration, simulation, asymptotic theory, central limit theorem, bootstrap, resampling
A sampling distribution describes how a statistic (like the sample mean or sample median) varies across every possible sample you could draw from a population. The key measures for evaluating an estimator are its bias, variance, and mean squared error (MSE). Because exhaustive enumeration is almost never feasible, we rely on mathematical derivation, asymptotic theory, simulation, or bootstrap resampling to approximate sampling distributions.
Sampling distribution
The probability distribution of a statistic (e.g. sample mean, sample proportion) as its values vary across all possible samples of size n drawn from a given population.
Statistic
Any function of the observed data, such as the sample mean, sample median, or sample standard deviation. A statistic is itself a random variable because it depends on which sample is drawn.
Estimator
A statistic used to estimate an unknown population parameter. For example, the sample mean is an estimator of the population mean.
Bias
The difference between the expected value of the estimator and the true parameter value: Bias = E[θ-hat] − θ. When bias equals zero, the estimator is called unbiased.
Unbiased estimator
An estimator whose expected value equals the parameter it estimates, so that on average it hits the target.
Mean squared error (MSE)
The average squared distance between the estimator and the true parameter: MSE(θ-hat) = E[(θ-hat − θ)²]. It decomposes neatly into variance plus squared bias: MSE = Var(θ-hat) + [Bias]².
Concentration
A measure of how tightly the estimator clusters around the true parameter value. Formally: P[|θ-hat − θ| < ε] for small ε.
Empirical distribution function (edf)
The sample-based estimate of the population CDF, placing mass 1/n on each observed data value: F-hat(x) = (number of observations ≤ x) / n.
The core question is: how well does an estimator θ-hat estimate the true parameter θ?
To answer this, we need to know how θ-hat behaves across all possible samples from the population. That behaviour is the sampling distribution.
If you could draw M samples of size n from the population, compute θ-hat from each, and plot the M values, you would be approximating the sampling distribution.
In practice, taking many repeated samples is almost always impossible due to cost, time, or logistics.
Concentration
How likely is θ-hat to land close to θ? Compute P[|θ-hat − θ| < ε] for small values of ε. Higher probability means a better estimator.
Bias
On average, does the estimator equal the target? Bias = E[θ-hat] − θ.
If Bias = 0, the estimator is unbiased.
A small bias is often acceptable if the variance is also small.
Mean squared error (MSE)
Captures both accuracy and precision in a single number:
MSE(θ-hat) = Var(θ-hat) + [Bias]²
An estimator with zero bias has MSE equal to its variance.
An estimator with some bias can still have lower MSE than an unbiased one if its variance is sufficiently smaller.
This is worth memorising:
MSE(θ-hat) = Var(θ-hat) + [Bias]²
It tells you that total estimation error comes from two sources: how spread out the estimator is (variance) and how systematically off-centre it is (bias). Reducing one sometimes increases the other.
When the population distribution is known, you can sometimes derive the exact sampling distribution analytically.
If X₁, …, Xₙ are iid N(μ, σ²), then the sample mean X-bar follows N(μ, σ²/n) exactly.
If X₁, …, Xₙ are iid N(μ, σ²), then (n−1)S²/σ² follows a chi-square distribution with df = n − 1.
If T₁, …, Tₙ are iid Exp(β), then nβ-hat = ΣTᵢ follows Gamma(n, β).
If Y₁, …, Yₙ are iid Bernoulli(p), then np-hat = ΣYᵢ follows Binomial(n, p).
When exact derivation is difficult, the Central Limit Theorem (CLT) and its variants provide approximate results for large samples.
The CLT applies to the sample mean, sample proportion, sample median, sample quantiles, and MLEs.
The larger n is, the better the approximation.
For a fixed n, the closer the population distribution is to symmetric with normal-like tails, the more accurate the normal approximation.
When you know the population distribution but cannot derive the sampling distribution analytically, you can simulate.
Specify the population distribution exactly (e.g. Cauchy with known parameters).
Generate many samples of size n, compute the statistic from each, and study the resulting collection.
Limitation: you must fully specify the population, which is rarely possible in practice.
Design considerations for a simulation study include choosing the sample sizes, parameter values, and number of replications.
When the population distribution is unknown and n is too small for asymptotic theory, the bootstrap provides an approximation.
Draw many samples of size n with replacement from the observed data.
Compute the statistic from each resample.
Use these resampled values to estimate the sampling distribution.
The bootstrap is covered in detail in Part 4 of these notes.
MSE decomposition: MSE(θ-hat) = Var(θ-hat) + [Bias]²
Bias: Bias = E[θ-hat] − θ
Concentration probability: P[|θ-hat − θ| < ε]
⚠️ The MSE decomposition (Var + Bias²) appears frequently in exam questions. Know how to apply it, not just state it.
⚠️ "Unbiased" does not mean "best." An estimator can be unbiased yet have enormous variance. MSE is the more complete measure.
⚠️ Be able to name the four methods for determining a sampling distribution and when each is appropriate.
⚠️ Know the exact sampling distributions: X-bar from normal data is normal; (n−1)S²/σ² is chi-square; sum of exponentials is gamma; sum of Bernoullis is binomial.
Q: What is the sampling distribution of a statistic?
A: The probability distribution of the statistic across all possible samples of a given size from the population.
Q: Write the MSE decomposition and explain each component.
A: MSE(θ-hat) = Var(θ-hat) + [Bias]². Variance measures the spread of the estimator around its own mean; squared bias measures how far that mean is from the true parameter.
Q: If X₁, …, Xₙ are iid N(μ, σ²), what is the exact distribution of the sample mean?
A: X-bar ~ N(μ, σ²/n).
Q: Why is simulation alone insufficient for most real-world problems?
A: Simulation requires you to specify the population distribution exactly, which is rarely known in practice. If the assumed distribution is wrong, the simulation results are unreliable.
Q: Under what conditions would you use bootstrap resampling rather than asymptotic theory?
A: When the population distribution is unknown and the sample size is too small for CLT-based approximations to be reliable.
sampling distribution, estimator bias, mean squared error, MSE decomposition, unbiased estimator, variance of estimator, CLT, central limit theorem, bootstrap, resampling, simulation study, mathematical derivation, concentration probability, chi-square distribution, gamma distribution, binomial distribution, empirical distribution function, edf, Tamhane Dunlop, Principles of Statistics, STAT Texas A&M, Handout 10