Difficulty: Intermediate | Prerequisites: Basic probability (random variables, expectation, variance), familiarity with common distributions (Bernoulli, Uniform, Normal).
Estimation is the bridge between probability theory and practical data analysis. Up to this point in the course, you have been given population parameters and asked to compute probabilities. Now the direction reverses: you observe data and try to recover the unknown parameters that generated it. This is the foundation for confidence intervals, hypothesis testing, and nearly every applied method that follows. If you skipped the lectures on expectation and variance, go back to those first.
An estimator is a function of your sample data that guesses an unknown population parameter. Good estimators are unbiased (correct on average), consistent (converge to the truth as your sample grows), and efficient (lowest variance among competitors). The sample mean is the workhorse example: it is unbiased and consistent for the population mean under very general conditions.
Estimator
A function of the observed sample data, θ̂ = θ̂(X₁, …, Xₙ), used to approximate an unknown population parameter θ. Think of it as a recipe that takes your data and spits out a best guess for the true value.
Bias
Bias(θ̂) = E_θ₀[θ̂] − θ₀. The systematic difference between what the estimator returns on average and the true parameter value. In simple terms, bias tells you whether your guesses are centred on the right answer or consistently off to one side.
Unbiased estimator
An estimator where E_θ₀[θ̂] = θ₀ for all values of θ₀. The estimator equals the true parameter only on average, not necessarily for any single sample. Think of it as: if you repeated the experiment thousands of times, the average of all your estimates would land on the true value.
Consistent estimator
An estimator that converges in probability to the true parameter as the sample size n increases. For all ε > 0, lim(n→∞) P(|θ̂ − θ₀| < ε) = 1. In simple terms, the more data you collect, the closer your estimate gets to the truth, with high probability.
Standard error
SE(θ̂) = √Var(θ̂(X₁, …, Xₙ)). The standard deviation of the estimator's sampling distribution. Think of it as: how much would your estimate bounce around if you kept drawing new samples of the same size?
Efficiency
A comparison between two unbiased estimators. θ̂_A is efficient relative to θ̂_B if Var(θ̂_B) ≥ Var(θ̂_A). In simple terms, the efficient estimator wastes less information, so it gives you tighter estimates from the same amount of data.
Sample mean (X̄ₙ)
X̄ₙ = (1/n) Σᵢ Xᵢ. The arithmetic average of the observed values. It is unbiased and consistent for the population mean μ.
Sample variance (Sₙ²)
Sₙ² = (1/(n−1)) Σᵢ (Xᵢ − X̄ₙ)². The average squared deviation from the sample mean, divided by n−1 (not n). The n−1 denominator corrects the bias that would otherwise make the estimator systematically underestimate the true variance.
An estimator θ̂ = θ̂(X₁, …, Xₙ) is itself a random variable, because it is a function of random data. Before you observe the data, the estimator has a distribution; after you observe it, you get a single number called the estimate.
The goal: choose an estimator that reliably recovers the unknown parameter θ from data.
X ~ Bernoulli(θ), so each observation is 0 or 1.
Natural estimator: the sample mean θ̂ = (1/n) Σ Xᵢ, which equals the proportion of successes.
Example: five trials give (1, 0, 0, 1, 1), so θ̂ = 3/5.
The sum S₅ = Σ Xᵢ follows a Binomial(5, θ₀) distribution. Dividing by 5 gives the estimator, which can take values in {0, 1/5, 2/5, 3/5, 4/5, 1}.
X ~ Uniform(0, θ), so the density is f(x) = 1/θ for 0 ≤ x ≤ θ.
The parameter θ is the upper boundary of the distribution. You need an estimator that recovers this boundary from the data.
Expectation: E_θ[X] = ∫ x · f_X(x | θ) dx
Variance: Var_θ(X) = ∫ (X − E_θ[X])² · f_X(x | θ) dx
Both depend on the unknown parameter θ, which is why estimating θ matters in the first place.
Bias(θ̂) = E_θ₀[θ̂] − θ₀. If this equals zero for every possible value of θ₀, the estimator is unbiased.
Unbiasedness means the estimator is correct on average. It does not mean every single estimate is close to the truth.
Example: for X₁, …, Xₙ i.i.d. from N(μ, σ²), the sample mean X̄ₙ satisfies E[X̄ₙ] = μ, so it is unbiased for μ.
The population variance is σ² = E[(X − E[X])²].
A first attempt, σ̂² = (1/n) Σ(Xᵢ − X̄ₙ)², is biased: it systematically underestimates σ².
The corrected version, Sₙ² = (1/(n−1)) Σ(Xᵢ − X̄ₙ)², is unbiased for σ². This is the empirical (sample) variance.
The algebraic expansion: (1/n) Σ(Xᵢ − X̄ₙ)² = (1/n) Σ Xᵢ² − X̄ₙ². Working through this shows why the 1/n version loses a degree of freedom and needs the n−1 correction.
θ̂ is consistent for θ if, as n → ∞, the estimator converges in probability to the true parameter: for all ε > 0, P(|θ̂ − θ₀| < ε) → 1.
Interpretation: in a large enough sample, a consistent estimator will be within any small distance of the truth with high probability.
The sample mean is a consistent estimator for μ when drawing i.i.d. from N(μ, σ²). This follows from the Law of Large Numbers.
SE(θ̂) = √Var(θ̂). It quantifies the spread of the estimator's sampling distribution.
A smaller standard error means the estimator is more precise, all else equal.
Compares two unbiased estimators for the same parameter.
θ̂_A is efficient relative to θ̂_B if Var(θ̂_B) ≥ Var(θ̂_A). That is, A has equal or lower variance.
The efficient estimator extracts more information from the same data. You always prefer the estimator with lower variance, assuming both are unbiased.
\hat{\theta} = \hat{\theta}(X_1, \ldots, X_n)\bar{X}_n = \frac{1}{n} \sum_{i=1}^{n} X_i\text{Bias}(\hat{\theta}) = E_{\theta_0}[\hat{\theta}] - \theta_0S_n^2 = \frac{1}{n-1} \sum_{i=1}^{n} (X_i - \bar{X}_n)^2\text{SE}(\hat{\theta}) = \sqrt{\text{Var}(\hat{\theta}(X_1, \ldots, X_n))}\hat{\theta}_A \text{ efficient relative to } \hat{\theta}_B \iff \text{Var}(\hat{\theta}_B) \geq \text{Var}(\hat{\theta}_A)Note on the variance expansion: (1/n) Σ(Xᵢ − X̄)² = (1/n) Σ Xᵢ² − 2X̄ · (1/n) Σ Xᵢ + X̄² = (1/n) Σ Xᵢ² − X̄². This is the algebraic identity used to show why dividing by n gives a biased estimator of σ².
Polling and surveys use the sample mean as an estimator for the true proportion of a population that holds a certain view. The standard error of that estimator is what news outlets report as the "margin of error."
In manufacturing quality control, engineers estimate the variance of a production process to determine whether parts are being made within acceptable tolerances. The sample variance Sₙ² is the standard tool for this.
Students often think "unbiased" means the estimator always equals the true parameter. It does not. Unbiased means the estimator equals the true parameter on average across many repeated samples. Any single estimate can be far off.
Students frequently confuse the estimator (a random variable, before observing data) with the estimate (a single number, after observing data). The estimator has a distribution; the estimate is just one realisation of it.
Dividing by n instead of n−1 when computing sample variance is one of the most common calculation errors. The 1/n version is biased. The correction to n−1 accounts for the fact that you estimated the mean from the same data.
Students sometimes assume consistency implies unbiasedness, or vice versa. These are independent properties. An estimator can be consistent but biased in finite samples, or unbiased but not consistent (rare, but possible with pathological constructions).
⚠️ You will almost certainly be asked to prove that the sample mean is unbiased for μ. Know the one-line proof: E[X̄ₙ] = (1/n) Σ E[Xᵢ] = (1/n)(nμ) = μ.
⚠️ Expect a question asking you to show that (1/n) Σ(Xᵢ − X̄)² is biased, and that dividing by n−1 corrects it. The algebraic expansion is a classic exam derivation.
⚠️ Know the definition of consistency in terms of convergence in probability. Be prepared to state it precisely with the limit and the ε.
⚠️ Efficiency comparisons between two estimators are a common short-answer item. You need to check that both are unbiased, then compare their variances.
True or False: An unbiased estimator always produces an estimate equal to the true parameter.
False. It equals the true parameter on average, not for every sample.
True or False: The sample variance Sₙ² divides by n−1, not n, to correct for bias.
True.
Fill in the blank: A consistent estimator converges in ______ to the true parameter as n → ∞.
Probability.
True or False: If θ̂_A has lower variance than θ̂_B and both are unbiased, then θ̂_A is more efficient.
True.
Fill in the blank: The standard error of an estimator is the ______ of its sampling distribution.
Standard deviation.
Q: Let X₁, …, Xₙ be i.i.d. Bernoulli(θ). Show that the sample mean is an unbiased estimator for θ.
A: E[X̄ₙ] = E[(1/n) Σ Xᵢ] = (1/n) Σ E[Xᵢ] = (1/n)(nθ) = θ. Since E[X̄ₙ] = θ for all θ, the sample mean is unbiased.
Q: Why does the sample variance use n−1 in the denominator instead of n?
A: Dividing by n produces a biased estimator that systematically underestimates σ². The denominator n−1 corrects for the fact that X̄ₙ is estimated from the same data, which uses up one degree of freedom. The corrected version Sₙ² = (1/(n−1)) Σ(Xᵢ − X̄ₙ)² satisfies E[Sₙ²] = σ².
Q: State the definition of a consistent estimator.
A: θ̂ is consistent for θ₀ if, for all ε > 0, lim(n→∞) P(|θ̂(X₁, …, Xₙ) − θ₀| < ε) = 1. In words, the probability that the estimator is within any small distance of the truth approaches 1 as the sample size grows.
Q: Suppose θ̂_A and θ̂_B are both unbiased for θ. Var(θ̂_A) = 4/n and Var(θ̂_B) = 9/n. Which is more efficient?
A: θ̂_A, because Var(θ̂_A) < Var(θ̂_B). It extracts more information per observation.
Q: For X₁, …, Xₙ i.i.d. from N(μ, σ²), is the sample mean consistent for μ? Why?
A: Yes. Var(X̄ₙ) = σ²/n → 0 as n → ∞, and X̄ₙ is unbiased for μ. By Chebyshev's inequality (or the Law of Large Numbers), X̄ₙ converges in probability to μ.
This material connects directly to confidence intervals: once you have an unbiased estimator and its standard error, you can build an interval that contains the true parameter with a specified probability. The properties covered here (bias, consistency, efficiency) are the criteria you will use to choose between estimators throughout the rest of the course.
Consistency ties back to the Law of Large Numbers from your probability unit. The convergence-in-probability definition here is the same concept, applied to a specific function of the data rather than to the raw sample average.
Maximum likelihood estimation (MLE), which likely comes next in the course, is a systematic method for constructing estimators. The properties you have learned here are exactly what you will use to evaluate whether an MLE is any good.
Estimation, estimator, estimate, point estimation, bias, unbiased estimator, consistent estimator, efficiency, relative efficiency, standard error, sampling distribution, sample mean, sample variance, empirical variance, Bessel's correction, n minus 1, degrees of freedom, convergence in probability, Law of Large Numbers, Bernoulli estimation, uniform distribution parameter estimation, normal distribution mean estimation, variance estimation, analytical statistics, NYU statistics