These notes cover what makes OLS estimators biased (and what does not), how simple and multiple regression estimates compare depending on the correlation structure among regressors, and what the Gauss-Markov theorem tells us about the efficiency of OLS relative to other linear unbiased estimators. The core thread: understanding when you can trust a regression coefficient and when adding or omitting variables changes your estimates.
Difficulty: Intermediate | Prerequisites: Introductory statistics, simple linear regression (Chapter 2), basic matrix/summation notation.
Chapter 3 moves from simple regression (one explanatory variable) to multiple regression, where several variables appear on the right-hand side. The central question shifts from "what is the slope?" to "what happens to the slope when I add or remove controls?" This is where omitted variable bias, multicollinearity, and the efficiency properties of OLS come into sharp focus.
You need to be comfortable with the mechanics of simple OLS from Chapter 2, including the algebra of the slope formula, the concept of residuals, and the first four Gauss-Markov assumptions. If those are shaky, revisit Chapter 2 before working through this material.
OLS estimator (ordinary least squares)
The slope and intercept values that minimise the sum of squared residuals. In simple regression, the OLS slope is the sample covariance of x and y divided by the sample variance of x.
In simple terms, OLS finds the line that fits the data as closely as possible, measured by the total squared vertical distance from each point to the line.
Omitted variable bias (OVB)
The distortion in a coefficient estimate that arises when a relevant variable is left out of the regression and that variable is correlated with an included regressor.
Think of it as: if you leave out something that matters, and it is tangled up with a variable you did include, the included variable's coefficient picks up the effect of both.
Multicollinearity
A situation where two or more independent variables in a multiple regression are highly (but not perfectly) correlated with each other.
In simple terms, your regressors are saying much the same thing, which makes it hard to tease apart their individual effects. It inflates standard errors but does not bias the coefficients.
Ceteris paribus
"All else equal." The interpretation of a partial regression coefficient: the estimated change in y for a one-unit change in x, holding all other included variables fixed.
Think of it as the thought experiment where you change one dial while freezing everything else.
Gauss-Markov assumptions
The set of conditions (linearity, random sampling, no perfect collinearity, zero conditional mean of errors, homoskedasticity) under which OLS is the best linear unbiased estimator (BLUE).
In simple terms, these are the rules the data and the model need to follow for OLS to be the most efficient unbiased estimator among all linear estimators.
Standard error of a coefficient, se(β̂)
An estimate of the standard deviation of the sampling distribution of a coefficient. Smaller standard errors mean more precise estimates.
Think of it as: how much would your estimate bounce around if you could re-draw the sample many times?
Heteroskedasticity
The condition where the variance of the error term changes across observations (e.g., the spread of residuals is larger for high-income individuals than low-income ones).
In simple terms, the noise in the data is not equally "loud" everywhere. It affects standard errors and efficiency but does not cause bias in the OLS coefficients themselves.
BLUE (best linear unbiased estimator)
The estimator with the smallest variance among all estimators that are both linear functions of the data and unbiased. The Gauss-Markov theorem proves OLS is BLUE under its assumptions.
Think of it as: OLS is the sharpest tool in the (linear, unbiased) toolbox.
Partialling out (Frisch-Waugh-Lovell)
A procedure where you regress x₁ on all other regressors, save the residuals, then regress y on those residuals. The resulting slope equals the multiple regression coefficient on x₁.
In simple terms, strip away everything the other variables can explain about x₁, then see what is left over and how it relates to y.
Tags: OLS bias, heteroskedasticity, omitted variable, multicollinearity, unbiasedness
The question is simple: which violations of the classical assumptions actually make OLS coefficients point to the wrong number on average?
Heteroskedasticity does not cause bias. It violates the homoskedasticity assumption (Gauss-Markov 5), which affects efficiency and standard errors, but the OLS coefficients remain unbiased. You lose BLUE status, not unbiasedness.
Omitting an important variable causes bias. If the omitted variable belongs in the true model and is correlated with an included regressor, the included variable's coefficient absorbs part of the omitted variable's effect. This is the textbook case of omitted variable bias.
High correlation between included regressors does not cause bias. A sample correlation of 0.95 between two included independent variables is multicollinearity. It inflates standard errors (makes estimates imprecise) but does not shift the expected value of the OLS estimator away from the true parameter. Unbiasedness is preserved; only efficiency suffers.
The upshot: of the three candidates, only omitting an important variable produces bias. This distinction between bias (a property of the expected value) and efficiency (a property of the variance) runs through the entire chapter.
Tags: simple regression, multiple regression, omitted variable bias direction, standard error comparison, control variables
This section works through the scenarios where adding control variables (x₂, x₃) to a regression of y on x₁ changes (or does not change) the coefficient on x₁. The concrete example: y = final exam score, x₁ = class attendance, x₂ = prior GPA, x₃ = SAT/ACT score.
x₁ highly correlated with x₂ and x₃, and x₂/x₃ have large effects on y: The simple regression estimate β̃₁ and the multiple regression estimate β̂₁ will be very different. The simple regression conflates the effect of attendance with the effects of GPA and test scores (because they move together). Adding controls strips out the confounding, so the estimate changes substantially.
x₁ nearly uncorrelated with x₂ and x₃ (even if x₂ and x₃ are correlated with each other): β̃₁ and β̂₁ will be similar. The omitted variable bias formula shows that what matters for the bias in β̃₁ is the correlation between x₁ and the omitted variables. If x₁ is orthogonal to x₂ and x₃, leaving them out barely shifts the estimate of β₁.
x₁ highly correlated with x₂/x₃, and x₂/x₃ have small effects on y: se(β̃₁) from the simple regression is likely smaller. Adding x₂ and x₃ does not reduce the error variance much (small partial effects), but it introduces multicollinearity with x₁, which inflates Var(β̂₁). The denominator of the variance formula for β̂₁ shrinks because of the high R² of x₁ on x₂ and x₃.
x₁ nearly uncorrelated with x₂/x₃, and x₂/x₃ have large effects on y (and are correlated with each other): se(β̂₁) from the multiple regression is likely smaller. Adding the controls soaks up a large portion of the error variance (because x₂ and x₃ strongly predict y), which pushes σ̂² down. Meanwhile, the near-zero correlation of x₁ with x₂/x₃ means Var(β̂₁) does not suffer from multicollinearity. The net effect is a tighter estimate.
Adding relevant control variables to a regression does two things that pull in opposite directions on the precision of β̂₁:
It reduces the error variance (σ²), which lowers Var(β̂₁). This helps.
It reduces the variation in x₁ that is "independent" of the other regressors (captured by the VIF), which raises Var(β̂₁). This hurts.
Which effect dominates depends on how much x₁ correlates with the controls and how much the controls explain y.
Tags: Gauss-Markov, BLUE, alternative estimator, Cauchy-Schwarz inequality, linear unbiased estimator, variance comparison
This section covers the theoretical heart of Chapter 3: proving that OLS has the smallest variance among all linear unbiased estimators.
Start with the simple regression y = β₀ + β₁x + u. Pick any function g(x), for instance g(x) = x² or g(x) = log(1 + x²), and define zᵢ = g(xᵢ). Then define:
β̃₁ = Σ(zᵢ - z̄)yᵢ / Σ(zᵢ - z̄)xᵢ
This uses z-weights instead of x-weights. It looks odd, but it turns out to be a valid estimator under the right conditions.
Because E[u|x] = 0, you can treat both xᵢ and zᵢ as fixed (nonrandom) in the derivation.
Linearity: β̃₁ is a linear function of the yᵢ values. The weights (zᵢ - z̄)/Σ(zᵢ - z̄)xᵢ are constants (functions of x only).
Unbiasedness: Substitute yᵢ = β₀ + β₁xᵢ + uᵢ into the formula. The β₀ terms vanish because Σ(zᵢ - z̄) = 0. The β₁ terms simplify because Σ(zᵢ - z̄)xᵢ cancels. You are left with E[β̃₁] = β₁.
Adding the homoskedasticity assumption (Var(u|x) = σ²), the variance works out to:
Var(β̃₁) = σ² · Σ(zᵢ - z̄)² / [Σ(zᵢ - z̄)xᵢ]²
Compare this with the OLS variance: Var(β̂₁) = σ² / Σ(xᵢ - x̄)².
The Cauchy-Schwarz inequality states:
[n⁻¹ Σ(zᵢ - z̄)(xᵢ - x̄)]² ≤ [n⁻¹ Σ(zᵢ - z̄)²] · [n⁻¹ Σ(xᵢ - x̄)²]
Note that you can replace (xᵢ - x̄) with xᵢ in the left-hand side because Σ(zᵢ - z̄)x̄ = x̄ · Σ(zᵢ - z̄) = 0.
Rearranging the inequality shows that Σ(zᵢ - z̄)² / [Σ(zᵢ - z̄)xᵢ]² ≥ 1 / Σ(xᵢ - x̄)².
Multiply both sides by σ², and you get Var(β̃₁) ≥ Var(β̂₁).
This holds for any choice of g(x), so OLS has the smallest variance among all estimators of this form. Since the class of linear unbiased estimators can be written in this way, OLS is BLUE.
OLS slope (simple regression):
β̂₁ = Σ(xᵢ - x̄)yᵢ / Σ(xᵢ - x̄)²
Alternative slope estimator (z-weighted):
β̃₁ = Σ(zᵢ - z̄)yᵢ / Σ(zᵢ - z̄)xᵢ, where zᵢ = g(xᵢ) for some function g
Variance of the OLS slope (simple regression, homoskedastic):
Var(β̂₁) = σ² / Σ(xᵢ - x̄)²
Variance of the alternative estimator:
Var(β̃₁) = σ² · Σ(zᵢ - z̄)² / [Σ(zᵢ - z̄)xᵢ]²
Variance of a coefficient in multiple regression:
Var(β̂ⱼ) = σ² / [SSTⱼ · (1 - Rⱼ²)]
where SSTⱼ = Σ(xⱼᵢ - x̄ⱼ)² is the total sample variation in xⱼ, and Rⱼ² is the R-squared from regressing xⱼ on all other independent variables. The term 1/(1 - Rⱼ²) is the variance inflation factor (VIF).
Cauchy-Schwarz inequality (as used in the Gauss-Markov proof):
[Σ(zᵢ - z̄)(xᵢ - x̄)]² ≤ [Σ(zᵢ - z̄)²] · [Σ(xᵢ - x̄)²]
Students often think heteroskedasticity causes OLS to be biased. It does not. Heteroskedasticity affects efficiency and makes the usual standard errors unreliable, but the OLS point estimates remain unbiased.
Students confuse multicollinearity with perfect collinearity. High correlation (even 0.95) between included regressors inflates standard errors but does not violate the "no perfect collinearity" assumption and does not bias anything. Perfect collinearity (one variable is an exact linear function of another) makes OLS impossible to compute.
Students sometimes assume that adding more control variables always makes the estimate of β₁ more precise. Adding controls can increase the VIF for x₁ if x₁ is correlated with those controls, which raises Var(β̂₁) even as the error variance falls. Precision depends on which effect dominates.
A common error in the Gauss-Markov proof is forgetting that you can drop x̄ from the covariance Σ(zᵢ - z̄)xᵢ because Σ(zᵢ - z̄) = 0. This simplification is what connects the Cauchy-Schwarz result to the variance comparison.
⚠️ "Which of these causes bias?" is a classic multiple-choice format. Know the list cold: omitted variables (yes), heteroskedasticity (no, affects efficiency), multicollinearity (no, affects precision).
⚠️ Be prepared to explain, in words, the direction of omitted variable bias using the formula: bias = β₂ · δ̃₁, where β₂ is the effect of the omitted variable on y and δ̃₁ is the coefficient from regressing the omitted variable on x₁.
⚠️ Questions comparing se(β̃₁) and se(β̂₁) test whether you understand the trade-off between lower error variance and higher VIF. Do not just memorise "more variables = more precise."
⚠️ The Gauss-Markov proof using Cauchy-Schwarz is a common problem-set and exam derivation. Practise writing it out from scratch, not just recognising it.
True or false: Heteroskedasticity makes OLS coefficients biased.
True or false: If x₁ is uncorrelated with x₂ and x₃, adding x₂ and x₃ to the regression will substantially change the estimated coefficient on x₁.
Fill in the blank: The Gauss-Markov theorem states that OLS is the best ______ ______ estimator under assumptions MLR.1 through MLR.5.
True or false: A high VIF for x₁ means the coefficient on x₁ is biased.
Fill in the blank: The variance of β̂₁ in simple regression equals σ² divided by ______.
Answers: 1. False. 2. False. 3. Linear unbiased. 4. False (imprecise, not biased). 5. Σ(xᵢ - x̄)².
Q: Name one violation of the classical assumptions that causes OLS estimators to be biased, and one that does not. Explain the difference.
A: Omitting a relevant variable that is correlated with an included regressor causes bias, because the included coefficient absorbs part of the omitted variable's effect. Heteroskedasticity does not cause bias; it only affects efficiency and invalidates the usual standard errors.
Q: In a regression of exam scores on attendance, GPA, and SAT, suppose attendance is highly correlated with GPA. Would you expect the simple regression coefficient on attendance (without GPA/SAT) to be close to the multiple regression coefficient? Why or why not?
A: They would likely differ substantially. In the simple regression, the attendance coefficient picks up part of the GPA effect (omitted variable bias). Adding GPA and SAT as controls strips out that confounding.
Q: Explain why adding control variables to a regression can either increase or decrease the standard error of the coefficient on x₁.
A: Adding controls reduces the error variance σ² (if the controls help explain y), which pushes se(β̂₁) down. But if x₁ is correlated with those controls, the VIF for x₁ rises, which pushes se(β̂₁) up. The net effect depends on which force is stronger.
Q: In the context of the Gauss-Markov theorem, what does BLUE stand for, and what does it tell you about OLS?
A: Best Linear Unbiased Estimator. It means that among all estimators that are (a) linear in y and (b) unbiased, OLS has the smallest variance. No other linear unbiased estimator can be more precise.
Q: Suppose you define an alternative estimator β̃₁ using weights based on zᵢ = xᵢ² instead of xᵢ. Under the Gauss-Markov assumptions, how does Var(β̃₁) compare with Var(β̂₁)?
A: Var(β̃₁) ≥ Var(β̂₁). The Cauchy-Schwarz inequality guarantees this. OLS uses the optimal weights (the xᵢ themselves), so any other weighting function produces a larger variance.
This material connects directly to hypothesis testing in Chapter 4: once you know the variance of your estimator, you can build t-statistics and confidence intervals. The trade-off between bias and variance reappears in model selection (Chapter 6 and beyond), where you weigh the cost of omitted variable bias against the cost of imprecision from including too many regressors.
The Gauss-Markov result also sets the stage for understanding when you might want to move beyond OLS. In later chapters, when homoskedasticity fails, weighted least squares (WLS) or robust standard errors become relevant. When the zero conditional mean assumption fails (endogeneity), instrumental variables (IV) replace OLS entirely.
OLS bias, omitted variable bias, OVB, heteroskedasticity vs bias, multicollinearity, variance inflation factor, VIF, simple vs multiple regression, ceteris paribus, partialling out, Frisch-Waugh-Lovell, Gauss-Markov theorem, BLUE, best linear unbiased estimator, Cauchy-Schwarz inequality, alternative slope estimator, standard error comparison, control variables, econometrics Chapter 3, Wooldridge Chapter 3, introductory econometrics, multiple regression properties