Difficulty: Intermediate | Prerequisites: Expected value, variance, law of large numbers (basic familiarity).
This set of notes covers the core properties used to evaluate estimators: whether they hit the target on average (bias), whether they zero in on the target as n grows (consistency), and how tightly they cluster around the target compared to alternatives (efficiency). The material draws on Problem 7 of the NYU Analytical Statistics Exam 2 practice set.
An estimator is a recipe for turning sample data into a guess about a population parameter. The three properties that matter most: does it hit the right target on average (unbiased), does it get closer to the target as the sample grows (consistent), and does it have the smallest possible spread among its competitors (efficient). These notes work through all three for concrete estimators of the population mean.
Unbiased estimator
An estimator theta-hat is unbiased for theta if E[theta-hat] = theta. On average, across all possible samples, it lands exactly on the true parameter. In simple terms: it does not systematically overshoot or undershoot.
Bias
Bias = E[theta-hat] - theta. A positive bias means the estimator overshoots on average; negative means it undershoots.
Consistent estimator
An estimator is consistent if it converges in probability to the true parameter as n grows. Two sufficient conditions: (1) bias goes to zero, and (2) variance goes to zero. Think of it as: with enough data, the estimator locks on to the right answer.
Efficient estimator
Among all unbiased estimators of a parameter, the most efficient one has the smallest variance. When comparing two unbiased estimators, the one with smaller variance is more efficient.
Mean squared error (MSE)
MSE = Var(theta-hat) + [Bias(theta-hat)]^2. This combines both spread and systematic error into a single measure of overall accuracy. For unbiased estimators, MSE equals the variance.
Sample variance with 1/n vs 1/(n-1)
Dividing the sum of squared deviations by n gives the MLE but is biased (it underestimates sigma^2). Dividing by n-1 gives the unbiased sample variance s^2.
Let X_1, ..., X_n be i.i.d. with mean mu and variance sigma^2. Three estimators of mu:
Estimator A: theta-hat_A = X-bar = (1/n) sum(X_i)
Estimator B: theta-hat_B = ((n-1)/n) X-bar
Estimator C: theta-hat_C = (X_1 + X_2) / 2
Estimator A:
E[theta-hat_A] = E[X-bar] = mu
Unbiased.
Estimator B:
E[theta-hat_B] = ((n-1)/n) E[X-bar] = ((n-1)/n) mu
This equals mu only if n-1 = n, which is never true. So theta-hat_B is biased.
Bias = ((n-1)/n) mu - mu = -mu/n
Estimator C:
E[theta-hat_C] = (E[X_1] + E[X_2]) / 2 = (mu + mu) / 2 = mu
Unbiased.
Conclusion: A and C are unbiased. B is biased (it systematically underestimates mu when mu > 0).
An estimator is consistent if it converges in probability to mu as n grows. This requires both bias going to zero and variance going to zero.
Estimator A:
Bias = 0 for all n.
Var(X-bar) = sigma^2/n, which goes to 0.
Consistent.
Estimator B:
Bias = -mu/n, which goes to 0 as n grows.
Var(theta-hat_B) = ((n-1)/n)^2 * sigma^2/n, which goes to 0.
Consistent (even though it is biased for any finite n).
Estimator C:
Bias = 0.
Var(theta-hat_C) = Var((X_1 + X_2)/2) = (Var(X_1) + Var(X_2))/4 = 2sigma^2/4 = sigma^2/2
The variance is sigma^2/2 for all n. It does not go to zero.
Not consistent. No matter how large n is, this estimator ignores all observations except the first two.
The unbiased estimators are A and C.
Var(theta-hat_A) = sigma^2/n
Var(theta-hat_C) = sigma^2/2
For any n >= 3, sigma^2/n < sigma^2/2, so A is more efficient than C. Even for n = 2, sigma^2/2 = sigma^2/2, so they are equally efficient (which makes sense, because when n = 2, X-bar = (X_1 + X_2)/2 = theta-hat_C).
Relative efficiency of C to A: Var(A)/Var(C) = (sigma^2/n) / (sigma^2/2) = 2/n. For n = 10, the relative efficiency is 0.2, meaning C uses only 20% of the available information.
No. This is the MLE of sigma^2, and it is biased.
E\left[\frac{1}{n} \sum_{i=1}^{n}(X_i - \bar{X})^2\right] = \frac{n-1}{n} \sigma^2The bias is -sigma^2/n. The estimator systematically underestimates the true variance because it uses X-bar (estimated from the same data) rather than the true mu in the squared deviations, effectively "using up" one degree of freedom.
The unbiased version divides by (n-1) instead of n:
s^2 = \frac{1}{n-1} \sum_{i=1}^{n}(X_i - \bar{X})^2This is Bessel's correction.
Students often assume that an unbiased estimator must also be consistent. Estimator C is a counterexample: it is unbiased but not consistent, because its variance does not shrink with n.
A biased estimator can be consistent. Estimator B shows this: its bias is -mu/n, which vanishes as n grows, and its variance also vanishes. So it converges to the truth despite being biased at every finite sample size.
Students sometimes think that dividing by n to estimate sigma^2 "makes sense because there are n observations." The reason for dividing by n-1 is that using X-bar (which was computed from the data) instead of the true mu costs one degree of freedom. This is Bessel's correction.
Efficiency is defined only among unbiased estimators (in the standard treatment). Comparing the variance of a biased estimator to an unbiased one is not an efficiency comparison; you would use MSE instead.
⚠️ "Which estimators are unbiased / consistent / efficient?" is a classic exam question format. Be ready to compute E[theta-hat] and Var(theta-hat) for any given estimator.
⚠️ The 1/n vs 1/(n-1) question about the sample variance is almost guaranteed to appear. Know that 1/n is biased and the bias is -sigma^2/n.
⚠️ Consistency requires both bias and variance to vanish. An estimator that uses only a fixed number of observations (like C) cannot be consistent.
⚠️ Relative efficiency = Var(better) / Var(worse) is a common follow-up. For X-bar vs (X_1 + X_2)/2, the answer is 2/n.
True or False: X-bar is both unbiased and consistent for mu.
Answer: True.
Fill in the blank: An estimator that uses only a fixed number of observations cannot be ______, because its variance does not go to zero.
Answer: consistent.
True or False: The MLE of sigma^2 (dividing by n) is unbiased.
Answer: False. It is biased; E[(1/n) sum(X_i - X-bar)^2] = (n-1)/n * sigma^2.
Fill in the blank: Among unbiased estimators, the one with the smallest ______ is the most efficient.
Answer: variance.
Q: Let theta-hat = 3X-bar. Is this unbiased for mu?
A: E[3X-bar] = 3mu. This equals mu only if mu = 0, so in general it is biased. The bias is 2mu.
Q: Suppose theta-hat_1 has variance 4/n and theta-hat_2 has variance 10/n. Both are unbiased. Which is more efficient, and what is the relative efficiency?
A: theta-hat_1 is more efficient. Relative efficiency of theta-hat_2 to theta-hat_1 = Var(theta-hat_1)/Var(theta-hat_2) = (4/n)/(10/n) = 0.4. Equivalently, theta-hat_1 achieves the same precision with only 40% as many observations.
Q: Is the median of X_1, ..., X_n a consistent estimator of the population mean when the distribution is normal?
A: Yes. For a normal distribution, the population median equals the population mean, and the sample median converges in probability to the population median. So it is consistent for mu. However, it is less efficient than X-bar (its variance is about (pi/2) * sigma^2/n ≈ 1.57 * sigma^2/n, compared to sigma^2/n for X-bar).
Q: Give an example of an estimator that is biased but consistent.
A: theta-hat_B = ((n-1)/n) X-bar. Its bias is -mu/n, which goes to zero, and its variance also goes to zero. So it is consistent despite being biased at every finite n.
Estimator properties are the theoretical backbone of everything else in this course. When you compute a confidence interval, the centre is an unbiased (or approximately unbiased) estimator, and the width depends on its variance. When you choose between two study designs (as in Problem 3), you are comparing the efficiency of the resulting estimators. The bias-variance trade-off reappears in regression (overfitting vs underfitting) and in machine learning.
Related Terms / Search Tags
unbiased estimator, biased estimator, consistency, efficient estimator, relative efficiency, mean squared error, MSE, bias-variance trade-off, Bessel's correction, sample variance, degrees of freedom, law of large numbers, convergence in probability, X-bar, sample mean, MLE, maximum likelihood estimator, NYU analytical statistics, ASTATS exam 2