Difficulty: Intermediate | Prerequisites: STAT 211 or equivalent (basic probability, distributions, expectation, variance, normal distribution).
Simple linear regression is the starting point for modelling the relationship between two variables. It sits at the foundation of STAT 212 and nearly every applied statistics course that follows. You will use it constantly in later chapters on multiple regression, ANOVA, and beyond. Before starting, you should be comfortable with means, variances, normal distributions, and the idea of a random variable. If those feel shaky, revisit your STAT 211 notes first.
Simple linear regression fits a straight line through data to describe how a response variable (Y) changes, on average, as a predictor variable (x) changes. The line is found by minimising the sum of squared vertical distances from the data points to the line. The three unknown parameters are the intercept, the slope, and the error variance.
Regression
A methodology for studying the relationship between two (or more) sets of variables. In simple terms, it is the statistical tool you use when you want to ask "how does Y change when x changes?"
Independent variable (x)
Also called the predictor, explanatory variable, covariate, regressor, or feature. This is the variable you treat as known or controlled. Think of it as the input you choose or observe, and then you watch what happens to Y.
Dependent variable (Y)
Also called the response or outcome variable. This is the variable you are trying to explain or predict. It is treated as random, even when x is fixed.
Simple linear regression model
The equation Y_i = β₀ + β₁x_i + ε_i, where i = 1, …, n. In simple terms, each observed Y is the sum of a straight-line component (β₀ + β₁x_i) and a random error (ε_i).
Intercept (β₀)
The expected value of Y when x = 0. Think of it as where the regression line crosses the vertical axis.
Slope (β₁)
The change in the average response when the independent variable increases by one unit. If β₁ = −5, the average Y drops by 5 for every 1-unit rise in x.
Error term (ε_i)
The unobserved random variable capturing everything the straight line misses: measurement noise, omitted factors, natural variability. Also called "noise."
Error variance (σ²)
The variance of each ε_i. It controls how tightly the data cluster around the true line. When σ² = 0, the relationship is perfectly deterministic.
Homoscedasticity
The assumption that every error term has the same variance σ². In simple terms, the spread of the data around the line does not change as x changes.
Least squares estimates (β̂₀, β̂₁)
The values of the intercept and slope that minimise the sum of squared vertical distances (residuals) between the observed data and the fitted line. Think of them as the "best-fitting" intercept and slope under the least-squares criterion.
Error sum of squares (SSE)
SSE = Σ(Y_i − Ŷ_i)². The total squared distance of the data from the fitted line. Smaller is better.
Predicted values (Ŷ_i)
Ŷ_i = β̂₀ + β̂₁x_i. The point on the fitted line corresponding to each observed x_i.
Residuals (e_i)
e_i = Y_i − Ŷ_i. The observed vertical gap between each data point and the fitted line. These stand in for the unobservable errors ε_i.
Regression begins with paired observations (x_i, Y_i) for i = 1, …, n. The goal is to describe how Y responds to changes in x.
Common pairings: animal age vs. weight, height vs. weight, temperature vs. reaction time, father's height vs. son's height.
When the relationship is approximately linear, we write y ≈ ax + b and seek the best a and b.
Some relationships are exact. Galileo's free-fall equation y = √(2x/g) gives the exact time y for an object dropped from height x under ideal conditions. In practice, repeated measurements at the same height yield slightly different times because of recording errors, air resistance, and so on.
In the deterministic case, every observation sits exactly on the curve.
In the statistical case, observations scatter around the curve, and regression models that scatter.
Galton's father–son height data is a classic statistical example. Sons whose fathers share the same height do not all have the same height, and the variation is not just measurement error; it reflects genuine biological variability. This is the kind of setting where regression earns its keep.
The model is:
Y_i = β₀ + β₁x_i + ε_i, for i = 1, …, n.
The four assumptions:
ε₁, …, ε_n are unobserved random variables.
They are independent of each other.
Each ε_i follows a N(0, σ²) distribution, with the same σ² for all i (homoscedasticity).
β₀, β₁, and σ² are unknown but fixed parameters.
From these assumptions it follows that E(Y_i) = β₀ + β₁x_i and Var(Y_i) = σ².
The expected value of Y in the sub-population where the predictor equals x is a linear function of x. This is the key interpretive sentence for exams.
β₁ is the change in the average response per one-unit increase in x.
If β₁ = 0.514 (Galton's data), a one-inch increase in father's height is associated with an average increase of 0.514 inches in son's height.
If β₁ = −5, the average response decreases by 5 for each unit x goes up.
Given data, we minimise the error sum of squares:
f(b₀, b₁) = Σ [Y_i − (b₀ + b₁x_i)]²
Take partial derivatives with respect to b₀ and b₁, set both to zero, and solve.
Slope estimator:
β̂₁ = Σ(x_i − x̄)(Y_i − Ȳ) / Σ(x_i − x̄)²
Intercept estimator:
β̂₀ = Ȳ − β̂₁x̄
Estimating σ²:
σ̂² = SSE / (n − 2)
We divide by n − 2 (not n) because we have estimated two parameters (β₀ and β₁). This makes σ̂² unbiased, in the same spirit as dividing by n − 1 for the sample variance.
Galton's data example (R output):
β̂₀ = 33.887, β̂₁ = 0.514
The fitted line is ŷ = 33.887 + 0.514x. For the population from which these data were drawn, a one-inch increase in father's height corresponds to an estimated average increase of 0.514 inches in son's height.
Fit the model and extract results:
fit <- lm(y ~ x)
summary(fit)
resid <- fit$residuals
predict <- fit$fitted.values
Regression is the workhorse of prediction in fields from economics (forecasting sales from advertising spend) to medicine (estimating blood pressure from age). Galton's own work on heredity launched the method, and the word "regression" itself comes from his observation that sons' heights "regress" toward the population mean.
Students often think β₁ tells you the change in Y for a single individual. It does not. It tells you the change in the average Y across a sub-population.
Students frequently confuse residuals (e_i, observable) with errors (ε_i, unobservable). Residuals estimate errors, but they are not the same thing.
Dividing SSE by n instead of n − 2 is a common slip. The denominator is n − 2 because two parameters have been estimated.
The x values are treated as fixed (or conditioned on) in the model, even if they were observed randomly. Students sometimes think x must be literally controlled by the experimenter; it need not be.
⚠️ You will almost certainly be asked to interpret the slope in context (e.g. "for every one-unit increase in x, the average Y changes by β̂₁ units").
⚠️ Know the four model assumptions cold. Exam questions often ask you to list them or identify which one is violated in a given scenario.
⚠️ Be able to derive or reproduce the least squares formulas, especially β̂₁ and β̂₀.
⚠️ Know why we divide by n − 2, not n, when estimating σ².
True or false: The slope β₁ represents the change in a single observation Y when x increases by 1. (False – it is the change in the average Y.)
True or false: Residuals and error terms are the same thing. (False – residuals estimate the unobservable errors.)
Fill in the blank: The denominator in the formula for σ̂² is ____. (n − 2)
True or false: Under the SLR model, Var(Y_i) depends on x_i. (False – Var(Y_i) = σ² for all i, by the homoscedasticity assumption.)
Fill in the blank: β̂₀ = Ȳ − ____. (β̂₁x̄)
Q: In Galton's data, β̂₁ = 0.514. Interpret this in context.
A: For the population of father–son pairs from which the data were collected, it is estimated that when a father's height increases by 1 inch, his son's height increases, on average, by approximately 0.514 inches.
Q: State the four assumptions of the simple linear regression model.
A: (1) The errors ε_i are unobserved random variables. (2) They are independent of each other. (3) Each ε_i ~ N(0, σ²), with the same σ² for all i (homoscedasticity). (4) β₀, β₁, and σ² are unknown but fixed parameters.
Q: Why do we divide SSE by n − 2 rather than n when estimating σ²?
A: Because we have estimated two parameters (β₀ and β₁) from the data. Dividing by n − 2 corrects for this and makes the estimator unbiased.
Q: Write the R command that produces the least squares fit for response y and predictor x.
A: summary(lm(y ~ x))
Q: A student claims "the residual IS the error." What is wrong with this?
A: The error ε_i is the true (unobservable) deviation of Y_i from the population regression line. The residual e_i is the observed deviation of Y_i from the fitted line. Residuals approximate the errors but are not identical to them, because the fitted line is itself an estimate.
This material connects directly to residual diagnostics and R² (covered next in these notes), and later to multiple linear regression where the model extends to more than one predictor. The assumption-checking techniques here carry over almost unchanged to every regression model you will meet in STAT 212.
simple linear regression, SLR, least squares, OLS, ordinary least squares, regression line, slope, intercept, error term, noise, residual, predicted value, fitted value, SSE, error sum of squares, sigma squared, homoscedasticity, constant variance, Gauss, Galton, regression to the mean, STAT 212, Principles of Statistics II, Texas A&M