Difficulty: Intermediate | Prerequisites: Basic probability, expected value, variance properties
An estimator is unbiased if its expected value equals the true parameter it is estimating. Bias and variance are separate qualities: an estimator can be unbiased but noisy, or biased but precise. When comparing two unbiased estimators, the one with lower variance is more efficient and generally preferred, because it clusters more tightly around the truth.
Unbiased estimator
An estimator whose expected value equals the true population parameter. Formally, E[θ̂] = θ. In simple terms, if you repeated your sampling procedure infinitely many times and averaged all the estimates, you would land exactly on the true value.
Bias
The systematic difference between an estimator's expected value and the true parameter: Bias = E[θ̂] − θ. Think of it as how far off-centre your estimates land on average, regardless of how spread out they are.
Variance of an estimator
A measure of how spread out the estimator's values are across repeated samples. Lower variance means the estimator gives more consistent results each time you draw a new sample.
Standard error
The standard deviation of an estimator's sampling distribution. For a sample mean of n independent observations with population standard deviation σ, the standard error is σ / √n.
Efficiency
When comparing two unbiased estimators of the same parameter, the one with smaller variance is called more efficient. In simple terms, it wastes less information from the sample.
Mean squared error (MSE)
MSE = Variance + Bias². This combines both sources of estimation error into a single number. For unbiased estimators, MSE equals the variance.
Compute E[θ̂] by applying linearity of expectation.
If E[θ̂] = θ (the true parameter), the estimator is unbiased.
Measurement error does not necessarily introduce bias. If the error has mean zero (like the faulty-scale problem in Problem 1, where F has E[F] = 0), the estimator stays unbiased even though the individual observations are noisier.
For n independent observations each with variance σ², the variance of the sample mean X̄ is σ² / n.
When observations come from different sources with different variances, you sum the individual variances and divide by n².
Independence matters: if two components of an observation are independent (e.g. true weight and scale error), Var(A_i) = Var(True weight) + Var(F_i).
Problem 1 setup: Hospital A's reported weight = true weight + random error F_i. Hospital B reports accurately.
Each A_i has Var(A_i) = Var(weight) + Var(F) = 1 + 1 = 2 (assuming independence between true weight and error).
Each B_i has Var(B_i) = 1.
X̄ = (ΣA_i + ΣB_i) / 200 remains unbiased because E[F] = 0, but Var(X̄) is larger than it would be without the measurement error.
Var(X̄) = [100 × Var(A_i) + 100 × Var(B_i)] / 200² = [100(2) + 100(1)] / 40,000 = 300/40,000 = 3/400.
Problem 1(c) proposes X̂ = (ΣA_i + 2 × ΣB_i) / 300. This over-weights the more reliable hospital (B) and down-weights the noisy one (A). Check bias first (still unbiased if E[X̂] = μ_W), then compare variance.
Problem 7 gives two estimators using different subsets of 256 observations: A uses 64, B uses 254 (note: the problem says 254, not 256). Both are unbiased (each is a sample mean of i.i.d. observations). B has lower variance because it uses more observations.
Even when both estimators are unbiased, the one with lower variance is preferred (more efficient).
B = Σ(i=1 to 254) x_i / 256. Note the mismatch: summing 254 observations but dividing by 256. This makes B slightly biased, since E[B] = 254 μ_x / 256 ≠ μ_x.
Both bias and variance matter when choosing between estimators. Use MSE = Bias² + Variance for a fair comparison.
If asked "which is unbiased," only A qualifies. If asked "which is closer to the truth," you need to compute MSE for both.
E[\bar{X}] = \mu \quad \text{(unbiased when E[error] = 0)}\text{Var}(\bar{X}) = \frac{1}{n^2} \sum_{i=1}^{n} \text{Var}(X_i)\text{For i.i.d. observations:} \quad \text{Var}(\bar{X}) = \frac{\sigma^2}{n}\text{Var}(A + B) = \text{Var}(A) + \text{Var}(B) \quad \text{(if A and B are independent)}\text{MSE}(\hat{\theta}) = \text{Bias}^2 + \text{Var}(\hat{\theta})\text{Var}(cX) = c^2 \text{Var}(X)Students often think measurement error always creates bias. It does not. If the error has mean zero, the estimator remains unbiased; the error only inflates the variance.
Students sometimes assume "more data is always better" without checking what that data contributes. An estimator using 254 observations divided by 256 is biased, even though it uses nearly all the data.
Confusing variance of individual observations with variance of the sample mean. The sample mean's variance shrinks with n; the individual observations' variance does not.
Assuming that the "better" estimator is always the unbiased one. When one estimator is unbiased with high variance and another is slightly biased with much lower variance, the biased one can have a smaller MSE and be more useful in practice.
⚠️ You will almost certainly be asked to verify whether a given estimator is unbiased. The procedure is always the same: compute E[θ̂] and check whether it equals θ.
⚠️ Variance of a sample mean when observations have different variances (e.g. some are noisier than others) is a recurring exam pattern. Do not default to σ²/n when the observations are not identically distributed.
⚠️ Comparing two estimators is a classic exam question. The expected sequence: check bias for both, compute variance for both, then state which is preferred and why.
⚠️ The Var(A + B) = Var(A) + Var(B) independence assumption is tested frequently. If the problem states independence, use it. If it does not, you cannot assume it.
True or false: An estimator that uses noisy data (data with random measurement error of mean zero) is always biased. (False. Mean-zero error preserves unbiasedness.)
Fill in the blank: The variance of the sample mean of n i.i.d. observations with variance σ² is ______. (σ²/n)
True or false: If estimator A is unbiased and estimator B is biased, A is always the better choice. (False. B could have lower MSE.)
Fill in the blank: Var(X + Y) = Var(X) + Var(Y) holds when X and Y are ______. (Independent)
True or false: Dividing by a number larger than your actual sample size makes an estimator biased downward. (True. E[sum/N] = n·μ/N, which is less than μ when n < N.)
Q: Hospital A reports baby weights with random error F (E[F] = 0, Var(F) = 1). Hospital B reports accurately. Both hospitals contribute 100 observations. Is X̄ = (ΣA_i + ΣB_i)/200 an unbiased estimator of μ_W?
A: Yes. E[A_i] = E[True weight] + E[F] = μ_W + 0 = μ_W. E[B_i] = μ_W. So E[X̄] = (100μ_W + 100μ_W)/200 = μ_W. The estimator is unbiased.
Q: What is Var(X̄) in the hospital problem above, given Var(weight) = 1 for both hospitals?
A: Var(A_i) = Var(weight) + Var(F) = 1 + 1 = 2 (by independence). Var(B_i) = 1. Var(X̄) = [100(2) + 100(1)] / 200² = 300/40,000 = 3/400 = 0.0075.
Q: A new estimator X̂ = (ΣA_i + 2·ΣB_i)/300 is proposed. Is it unbiased? Is it better than X̄?
A: E[X̂] = (100μ_W + 200μ_W)/300 = μ_W. It is unbiased. Var(X̂) = [100(2) + 4·100(1)] / 300² = 600/90,000 = 1/150 ≈ 0.00667. Since 1/150 < 3/400 (i.e. 0.00667 < 0.0075), X̂ has lower variance. X̂ is the better estimator.
Q: You have 256 observations. Estimator A = Σ(i=1 to 64) x_i / 64 and estimator B = Σ(i=1 to 254) x_i / 256. Which is unbiased?
A: E[A] = 64μ/64 = μ. Unbiased. E[B] = 254μ/256 ≠ μ. Biased (biased downward by a factor of 254/256).
Q: With σ_x = 8 and μ_x = 40, which estimator (A or B from above) has a higher probability of being within 1 of μ_x?
A: Var(A) = 64/64² = 1, so SE(A) = 1. P(|A − 40| < 1) = P(|Z| < 1) ≈ 0.6827. For B: E[B] = 254(40)/256 = 39.6875, Var(B) = 254(64)/256² = 0.248. SE(B) ≈ 0.498. P(|B − 40| < 1) = P(38.6875/0.498 < ... ) requires computing P(39 < B < 41). Standardise using E[B] = 39.6875: P((39 − 39.6875)/0.498 < Z < (41 − 39.6875)/0.498) = P(−1.38 < Z < 2.64) ≈ 0.912. B is more likely to be within 1 of μ_x despite being biased, because its variance is much smaller.
Estimator properties connect directly to confidence intervals: the width of a confidence interval depends on the estimator's standard error, which is the square root of its variance. A more efficient estimator gives a tighter interval. These concepts also feed into hypothesis testing, where the test statistic is built from the estimator's sampling distribution. Understanding bias and variance here is prerequisite to grasping Type I/II errors and power calculations.
Related Terms / Search Tags
Unbiased estimator, bias of estimator, variance of sample mean, standard error, efficiency, mean squared error, MSE, measurement error, sampling distribution, estimator comparison, Var(X-bar), E(X-bar), linearity of expectation, independent random variables, ASTATS NYU, analytical statistics midterm 2