Difficulty: Intermediate | Prerequisites: Sampling distributions, confidence intervals, chi-square distribution (see companion notes).
Simple linear regression is the workhorse of applied statistics. These notes cover how the ordinary least squares (OLS) estimators are derived, why they are unbiased, how to quantify their uncertainty, and how to interpret them in practice, including the log-log regression setting.
Ordinary least squares fits a straight line through data by minimising the sum of squared vertical distances between observations and the line. The slope and intercept estimators are unbiased under the standard assumptions, and their precision is measured by standard errors derived from the residual variance. Hypothesis tests and confidence intervals for the coefficients tell you whether the linear relationship is statistically meaningful and how strong it is.
Mean function E(Y|X)
The expected value of the response Y at a given value of the predictor X. In simple linear regression, this is modelled as a straight line: E(Y|X = x) = β₀ + β₁x. Think of it as: the average Y you would see if you could repeat the experiment many times at the same x.
Ordinary least squares (OLS)
The method that estimates regression coefficients by minimising the sum of squared residuals. In simple terms, this means: find the line that makes the total squared vertical miss as small as possible.
Slope (β̂₁)
The estimated change in the response for a one-unit increase in the predictor. Think of it as: the steepness of the fitted line.
Intercept (β̂₀)
The estimated value of the response when the predictor equals zero. In simple terms, this means: where the fitted line crosses the y-axis. It may or may not have a sensible real-world interpretation depending on whether x = 0 is within the range of the data.
Residual (eᵢ)
The difference between the observed response and the fitted value: eᵢ = Yᵢ – Ŷᵢ. Think of it as: what the model failed to explain for observation i.
Residual sum of squares (RSS)
The sum Σeᵢ², which measures total unexplained variation. OLS minimises this quantity.
Residual variance estimate (σ̂²)
RSS divided by (n – 2), the unbiased estimator of the error variance σ² in simple linear regression. The denominator is n – 2 because two parameters (β₀ and β₁) have been estimated. Think of it as: the average squared residual, corrected for the degrees of freedom used up by the line.
Standard error of an estimator
The estimated standard deviation of the sampling distribution of that estimator. In simple terms, this means: how much the coefficient estimate would vary if you drew new samples from the same population.
Unbiased estimator
An estimator whose expected value equals the parameter it targets: E(β̂) = β. Think of it as: on average, across many repetitions, the estimate lands on the truth.
The mean function E(Y|X = x) describes how the average response changes with the predictor.
Before fitting a model, always plot Y against X. A scatterplot reveals the overall trend, spread, outliers, and whether a straight line is a reasonable summary.
Estimating the mean function "by eye" means sketching a smooth curve through the cloud of points that captures the central tendency at each region of x. This is useful as a sanity check against the fitted regression line.
The slope estimator: β̂₁ = SXY / SXX, where SXY = Σ(xᵢ – x̄)(Yᵢ – Ȳ) and SXX = Σ(xᵢ – x̄)².
The intercept estimator: β̂₀ = Ȳ – β̂₁x̄.
The fitted line always passes through the point (x̄, Ȳ).
The proof relies on the fact (shown separately) that β̂₁ is unbiased: E(β̂₁) = β₁.
Take expectations: E(β̂₀) = E(Ȳ – β̂₁x̄) = E(Ȳ) – x̄ E(β̂₁).
Since E(Ȳ) = β₀ + β₁x̄ (the mean of the Yᵢ under the model) and E(β̂₁) = β₁, we get E(β̂₀) = β₀ + β₁x̄ – β₁x̄ = β₀.
The intercept estimator is therefore unbiased.
Var(β̂₁) = σ² / SXX. The standard error is SE(β̂₁) = σ̂ / √SXX.
Var(β̂₀) = σ²(1/n + x̄²/SXX). The standard error follows by taking the square root and replacing σ with σ̂.
These standard errors estimate the variability of the coefficient estimates across repeated samples. They tell you how precisely the slope and intercept have been estimated.
A 95% CI for β₁ is β̂₁ ± t*(n−2) × SE(β̂₁), where t* is the 97.5th percentile of the t-distribution with n – 2 degrees of freedom.
The same structure applies for β₀: β̂₀ ± t*(n−2) × SE(β̂₀).
Interpretation: if the CI for the slope does not contain zero, there is evidence of a linear association at the 5% level.
H₀: β₁ = 0 (no linear relationship). The test statistic is t = β̂₁ / SE(β̂₁), compared to a t(n – 2) distribution.
H₀: β₀ = 0 (the regression line passes through the origin). The test statistic is t = β̂₀ / SE(β̂₀).
Testing β₁ = 0 asks whether the predictor is useful. Testing β₀ = 0 asks whether the response is zero when the predictor is zero, which may or may not be scientifically meaningful.
σ̂² = RSS / (n – 2) = Σeᵢ² / (n – 2).
This estimates the variance of the errors around the true regression line. A smaller σ̂² means the observations cluster more tightly around the fitted line.
The denominator is n – 2 (not n or n – 1) because fitting the line consumes two degrees of freedom.
OLS slope estimator:
\hat{\beta}_1 = \frac{S_{XY}}{S_{XX}} = \frac{\sum (x_i - \bar{x})(Y_i - \bar{Y})}{\sum (x_i - \bar{x})^2}OLS intercept estimator:
\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{x}Variance of the slope:
\text{Var}(\hat{\beta}_1) = \frac{\sigma^2}{S_{XX}}Variance of the intercept:
\text{Var}(\hat{\beta}_0) = \sigma^2 \left(\frac{1}{n} + \frac{\bar{x}^2}{S_{XX}}\right)Residual variance estimator:
\hat{\sigma}^2 = \frac{\text{RSS}}{n - 2} = \frac{\sum e_i^2}{n - 2}Confidence interval for a coefficient:
\hat{\beta}_j \pm t^*_{n-2} \times \text{SE}(\hat{\beta}_j)t-test statistic:
t = \frac{\hat{\beta}_j}{\text{SE}(\hat{\beta}_j)}The Big Mac Index regression (log(bigmac2009) on log(bigmac2003)) is a classic example of a log-log model. In this setting, the slope β̂₁ is interpreted as an elasticity: a 1% increase in the 2003 Big Mac price is associated with roughly a β̂₁% change in the 2009 price. If β₁ ≈ 1, prices have grown proportionally across countries. If β₁ < 1, cheaper countries caught up; if β₁ > 1, expensive countries pulled further ahead.
The intercept β̂₀ in a log-log model represents the log of the predicted 2009 price when log(2003 price) = 0, i.e. when the 2003 price is $1. Testing β₀ = 0 asks whether a country with a $1 Big Mac in 2003 would also have a $1 Big Mac in 2009 (no systematic shift in price level).
Students often think the intercept always has a meaningful interpretation. It does not. If x = 0 is outside the range of the data, the intercept is just the mathematical anchor for the line, not a real prediction.
Students sometimes divide RSS by n or n – 1 instead of n – 2. In simple linear regression, two parameters are estimated, so the correct denominator for the unbiased variance estimate is n – 2.
Students frequently confuse the standard error of a coefficient with the residual standard error (σ̂). The former measures precision of the coefficient estimate; the latter measures the scatter of observations around the fitted line.
Students often assume a non-significant slope means there is no relationship. It means you lack evidence of a linear one at that sample size. The relationship could be nonlinear, or the sample could simply be too small.
⚠️ Proving unbiasedness of β̂₀ is a standard exam derivation. Memorise the three-step chain: write β̂₀ = Ȳ – β̂₁x̄, take expectations, substitute E(Ȳ) = β₀ + β₁x̄.
⚠️ You will likely be asked to interpret coefficients in a log-log regression. Remember: the slope is an elasticity (% change in Y per % change in X).
⚠️ Know the difference between n – 1 and n – 2 in the denominator of variance estimates. One parameter estimated (just the mean) gives n – 1; two (intercept + slope) gives n – 2.
⚠️ Be ready to explain what the standard errors of β̂₀ and β̂₁ are estimating, in plain language: the variability of those estimates across hypothetical repeated samples.
True or False: The OLS fitted line always passes through (x̄, Ȳ). Answer: True.
Fill in the blank: The denominator in σ̂² for simple linear regression is ___. Answer: n – 2.
True or False: If the 95% CI for β₁ contains zero, we reject H₀: β₁ = 0. Answer: False. We fail to reject.
Fill in the blank: In a log-log regression, the slope is interpreted as a(n) ___. Answer: Elasticity.
True or False: The intercept estimator is biased if the slope estimator is biased. Answer: True. The proof of unbiasedness for β̂₀ depends on E(β̂₁) = β₁.
Q: Show that the OLS intercept estimator β̂₀ is unbiased, given that β̂₁ is unbiased.
A: β̂₀ = Ȳ – β̂₁x̄. Take expectations: E(β̂₀) = E(Ȳ) – x̄ E(β̂₁). Under the model, E(Ȳ) = β₀ + β₁x̄ and E(β̂₁) = β₁. Substituting: E(β̂₀) = β₀ + β₁x̄ – β₁x̄ = β₀.
Q: You fit a simple linear regression of log(bigmac2009) on log(bigmac2003) and obtain β̂₁ = 0.87. Interpret this coefficient.
A: A 1% increase in the 2003 Big Mac price is associated with approximately a 0.87% increase in the 2009 Big Mac price. Since the elasticity is less than 1, countries with higher 2003 prices saw relatively smaller price increases by 2009, suggesting some convergence.
Q: The regression output reports SE(β̂₁) = 0.05 and SE(β̂₀) = 0.12. What are these quantities estimating?
A: They estimate the standard deviations of the sampling distributions of β̂₁ and β̂₀ respectively. If the study were repeated many times, SE(β̂₁) = 0.05 says the slope estimates would have a standard deviation of about 0.05.
Q: Construct a 95% CI for β₁ if β̂₁ = 0.87, SE(β̂₁) = 0.05, and t*₀.₀₂₅ with the appropriate df is approximately 2.00.
A: CI = 0.87 ± 2.00 × 0.05 = (0.77, 0.97). Since the interval does not contain 1, there is evidence that the elasticity is significantly less than 1.
Q: Why is testing H₀: β₀ = 0 of interest in the Big Mac regression?
A: In the log-log model, β₀ = 0 would mean that a country with a $1 Big Mac in 2003 is predicted to have a $1 Big Mac in 2009 (no systematic shift in price level). Rejecting this suggests a baseline change in Big Mac prices across the period.
Q: What does σ̂² represent and how is it computed?
A: σ̂² = RSS/(n – 2) is the unbiased estimate of the error variance σ², representing the average squared distance of observations from the true regression line. The denominator is n – 2 because fitting the line uses two degrees of freedom.
This connects to multiple regression (Chapter 3+) because the same OLS logic extends to more than one predictor, with matrix algebra replacing the scalar formulas. The residual degrees of freedom become n – p – 1 where p is the number of predictors.
The chi-square and t-distribution results from the companion sampling distributions notes underpin every confidence interval and hypothesis test here. The t-test for a coefficient is just the estimate divided by its standard error, compared to the same t-distribution.
Log transformations appear throughout applied regression whenever the relationship is multiplicative rather than additive, or when the data span several orders of magnitude.
simple linear regression, OLS, ordinary least squares, slope, intercept, beta-hat, fitted line, residual, RSS, residual sum of squares, sigma-hat-squared, error variance, standard error of slope, standard error of intercept, confidence interval for regression coefficient, t-test regression, unbiased estimator, mean function, scatterplot, log-log regression, elasticity, Big Mac index, Weisberg Applied Linear Regression, STAT 5302, Chapter 2