Source: Principles of Statistics I (Texas A&M), Tamhane/Dunlop Ch. 10–11
Tags: simple linear regression, SLR, least squares, regression model, deterministic relationship, non-deterministic relationship, response variable, explanatory variable, dependent variable, independent variable, error term, residuals, normal equations, SSxx, SSyy, SSxy, beta hat, slope, intercept, LSE, unbiased estimator
Simple linear regression models the mean of a response variable Y as a straight-line function of an explanatory variable X, plus normally distributed random error. The least squares method finds the line that minimises the total squared distance between observed and predicted values. The resulting estimators for slope and intercept are unbiased, normally distributed, and have known variances tied to the error variance and the spread of the X values.
Deterministic relationship
A relationship where knowing x completely determines y. Expressed as y = h(x), with no randomness involved.
Non-deterministic (random) relationship
A relationship where y depends on x but also includes a random component: y = h(x) + ε. For any fixed x, there is a whole population of possible y values.
Response variable (Y) / dependent variable
The outcome being measured or predicted. In the dose-strength example, this is the protective strength of the drug.
Explanatory variable (X) / independent variable
The variable used to predict or explain Y. In the dose-strength example, this is the drug dose level.
Simple linear regression model (SLR)
Y = β₀ + β₁X + ε, where ε ~ N(0, σ²). The mean of Y given X is a straight line, and the variance of Y is constant across all X values.
Error term (ε)
The random deviation of an observed Y from the true regression line. Assumed independent, normally distributed with mean 0 and constant variance σ².
Residual (eᵢ)
The observed error for data point i: eᵢ = yᵢ − ŷᵢ. The difference between what you actually observed and what the fitted line predicts.
Least squares estimators (LSE)
The values of β̂₀ and β̂₁ that minimise the sum of squared residuals (SSE). They are the "best-fitting" intercept and slope.
Sum of squares (SSxx, SSyy, SSxy)
Intermediate quantities used to compute the LSE. SSxx measures spread of X, SSyy measures spread of Y, SSxy measures how X and Y co-vary.
Normal equations
The pair of simultaneous equations obtained by setting the partial derivatives of SSE (with respect to β₀ and β₁) equal to zero. Solving them gives the least squares estimates.
A deterministic relationship leaves no uncertainty: if you know x, you know y exactly. The textbook example is apartment rental cost: y = 1000 + 750x, where x is the number of months.
A non-deterministic relationship adds randomness. If x is the lease length and y is total cost across all apartments in a city, then for a fixed x there is a wide spread of y values depending on apartment size, location, and so on. The model becomes y = h(x) + ε.
Regression analysis is the systematic approach to approximating h(·) and the distribution of ε.
The simplest regression model uses a straight line for h(x) and a normal distribution for ε:
Yᵢ = β₀ + β₁xᵢ + εᵢ, for i = 1, …, n
This requires four conditions to hold:
C1 – Normality: for each value of x, the population of y values is normally distributed
C2 – Constant variance (homoscedasticity): σ²(y|x) = σ² for all x. The spread of y does not change as x changes
C3 – Linear mean: μ(y|x) = β₀ + β₁x. The average value of y changes with x only through a straight line
C4 – Independence: the n data pairs (xᵢ, yᵢ) are independent of one another
The key implication: the distribution of Y is connected to X only through the mean, not through the variance.
The goal is to find the line ŷ = β̂₀ + β̂₁x that makes the sum of squared residuals as small as possible.
SSE (sum of squared errors):
Q(β₀, β₁) = SSE = Σ(yᵢ − β̂₀ − β̂₁xᵢ)²
Taking partial derivatives with respect to β₀ and β₁ and setting them to zero produces the normal equations:
nβ̂₀ + (Σxᵢ)β̂₁ = Σyᵢ
(Σxᵢ)β̂₀ + (Σxᵢ²)β̂₁ = Σxᵢyᵢ
Slope:
β̂₁ = SSxy / SSxx
Intercept:
β̂₀ = ȳ − β̂₁x̄
Where the building-block sums are:
SSxx = Σ(xᵢ − x̄)² = Σxᵢ² − (1/n)(Σxᵢ)²
SSyy = Σ(yᵢ − ȳ)² = Σyᵢ² − (1/n)(Σyᵢ)²
SSxy = Σ(xᵢ − x̄)(yᵢ − ȳ) = Σxᵢyᵢ − (1/n)(Σxᵢ)(Σyᵢ)
Estimator of σ:
σ̂ = √(SSE / (n − 2))
The denominator is n − 2 because two parameters (β₀ and β₁) have been estimated.
Unbiasedness:
E[β̂₁] = β₁
E[β̂₀] = β₀
Both estimators hit the true parameter on average.
Variances:
Var(β̂₁) = σ² / SSxx
Var(β̂₀) = σ²(1/n + x̄² / SSxx)
The variance of the slope estimator shrinks when SSxx is large, meaning a wider spread of X values gives you a more precise slope estimate.
Sampling distributions:
Both β̂₀ and β̂₁ are linear combinations of the yᵢ values (which are independent normals), so both are themselves normally distributed.
They are not independent of each other. Their covariance is:
Cov(β̂₀, β̂₁) = −σ² · x̄ / SSxx
Estimated standard errors:
SE(β̂₁) = σ̂ / √SSxx
SE(β̂₀) = σ̂ · √(1/n + x̄² / SSxx)
The quantity (n − 2)σ̂² / σ² follows a chi-squared distribution with df = n − 2 and is independent of both β̂₀ and β̂₁. This is why we can use t-distributions for inference.
A pharmaceutical firm tests 5 dose levels (0, 4, 8, 16, 32 mg) with 3 test tubes each (n = 15). Response y measures protective strength of the drug.
Computed summary statistics:
SSxx = 1785.6
SSyy = 764.4
SSxy = 1027.2
x̄ = 12.4, ȳ = 15.8
Results:
β̂₁ = 1027.2 / 1785.6 = 0.5753
β̂₀ = 15.8 − (0.5753)(12.4) = 8.67
SSE = 764.4 − (0.5753)(1027.2) = 173.45
σ̂ = √(173.45 / 13) = 3.653
Fitted line: ŷ = 8.67 + 0.5753x
Quantity | Formula |
|---|---|
SLR model | Yᵢ = β₀ + β₁xᵢ + εᵢ |
Slope estimate | β̂₁ = SSxy / SSxx |
Intercept estimate | β̂₀ = ȳ − β̂₁x̄ |
SSE | Σ(yᵢ − ŷᵢ)² = SSyy − β̂₁ · SSxy |
σ̂ | √(SSE / (n − 2)) |
Var(β̂₁) | σ² / SSxx |
Var(β̂₀) | σ²(1/n + x̄² / SSxx) |
⚠️ The four conditions (C1–C4) are constantly tested. You need to know each one and how to check it.
⚠️ The denominator in σ̂ is n − 2, not n − 1 or n. Losing two degrees of freedom corresponds to estimating two parameters (β₀ and β₁).
⚠️ A wider spread of X values (larger SSxx) produces more precise estimates of the slope. This is a common conceptual question.
⚠️ Constant variance means the scatter around the line is the same width everywhere along X. If it fans out, the condition is violated.
⚠️ SSE can also be computed as SSyy − β̂₁ · SSxy. This shortcut formula appears frequently in exam calculations.
Q: What are the four conditions required for the simple linear regression model?
A: C1, normality of Y for each X; C2, constant variance of Y across all X; C3, the mean of Y is a linear function of X; C4, the n observations are independent.
Q: Why do we divide SSE by (n − 2) rather than (n − 1) when estimating σ²?
A: Because two parameters (β₀ and β₁) have been estimated from the data, consuming two degrees of freedom.
Q: In the dose-strength example, calculate the fitted value ŷ when x = 10.
A: ŷ = 8.67 + 0.5753(10) = 14.42.
Q: If SSxx doubles (while σ² stays the same), what happens to Var(β̂₁)?
A: It halves. Var(β̂₁) = σ² / SSxx, so doubling SSxx cuts the variance in half.
Q: Are β̂₀ and β̂₁ independent? Explain.
A: No. Their covariance is −σ² · x̄ / SSxx. They are only independent when x̄ = 0.
simple linear regression, SLR model, least squares estimation, LSE, ordinary least squares, OLS, regression line, best fit line, slope, intercept, beta-zero, beta-one, residual, error term, epsilon, normal equations, SSE, SSxx, SSyy, SSxy, homoscedasticity, constant variance, unbiased estimator, sampling distribution of slope, degrees of freedom n minus 2, deterministic vs random relationship, response variable, explanatory variable, dependent variable, independent variable, regression assumptions, model conditions