Difficulty: Intermediate | Prerequisites: STAT 211 or equivalent (basic probability, expectation, variance, normal distribution)
Source: Lecture Notes, Chapter 1 (Part A) – Sharmistha Guha, Texas A&M University
Tags: simple linear regression, SLR, least squares, OLS, regression model, slope, intercept, predictor, response, dependent variable, independent variable, covariate, regression to the mean, Galton, scatterplot, line of best fit, STAT 212
Simple linear regression is the foundational tool for modelling the relationship between two quantitative variables. It sits at the very start of STAT 212 because nearly every more advanced method in the course (multiple regression, ANOVA, logistic regression) builds directly on the ideas introduced here. You should already be comfortable with expectation, variance, the normal distribution, and basic calculus (partial derivatives). If the phrase "minimise a function by setting its derivative to zero" feels unfamiliar, revisit your calculus notes before going further.
Simple linear regression models the average value of a response variable Y as a straight-line function of a single predictor x, plus random noise. The "best" line is found by least squares, which minimises the total squared vertical distance between the observed data points and the line. The method gives closed-form formulas for the estimated intercept and slope.
Regression
A methodology for studying the relationship between two (or more) sets of variables. In simple terms, regression asks: "How does one quantity change when another changes?"
Independent variable (x)
The variable whose values are treated as fixed or controlled. Also called the predictor, explanatory variable, covariate, regressor, or feature. Think of it as the input you choose or observe first.
Dependent variable (Y)
The variable whose values you are trying to explain or predict. Also called the response or outcome variable. It is random even when the corresponding x is fixed, because noise is always present.
Simple linear regression model
The equation Yi = β₀ + β₁xᵢ + εᵢ, for i = 1, …, n. It says each observed response equals a straight-line function of x plus a random error term.
Intercept (β₀)
The expected value of Y when x = 0. It anchors the regression line vertically.
Slope (β₁)
The change in the average response when the independent variable increases by one unit. If β₁ = −5, a one-unit increase in x corresponds to an average decrease of 5 in Y.
Error term (εᵢ)
An unobserved random variable representing the noise or deviation of the ith observation from the true line. Also called the disturbance.
Least squares
A method for estimating β₀ and β₁ by finding the values that minimise the sum of squared vertical distances between the observed data points and the fitted line. Carl Friedrich Gauss is credited with its discovery.
Least squares estimates (β̂₀ and β̂₁)
The specific values of the intercept and slope that minimise the sum of squared residuals. They are computed from the data using closed-form formulas.
Regression to the mean
The phenomenon (first observed by Sir Francis Galton) that extreme values of a predictor tend to produce less extreme values of the response. Galton noticed that very tall fathers had sons who were tall but closer to the population average, and the reverse for short fathers.
Regression studies how y changes when x increases or decreases. Typical examples:
Animal age (x) and animal weight (y)
Height (x) and weight (y)
Age (x) and blood pressure (y)
Temperature (x) and chemical reaction time (y)
Sometimes a theoretical equation links x and y exactly (e.g. the gravitational free-fall formula y = √(2x/g)). In practice, measurements include noise, so even a perfect physical law produces scattered data. Regression is the tool for recovering the underlying relationship from noisy observations.
Under ideal conditions, a physics equation may be exact: every repetition at the same height produces the same fall time. Real data never behave this way. When you drop a ball from the same height three times, you get three slightly different recorded times. Those differences arise from random measurement error (noise).
In Galton's father–son height data, the variability among sons of equally tall fathers is not solely measurement error. Genetics, nutrition, and many other factors contribute. This is an important conceptual distinction: noise in regression can come from measurement error, from omitted variables, or from inherent randomness in the system.
The simple linear regression model is:
Yi = β₀ + β₁xᵢ + εᵢ, i = 1, …, n
Four assumptions underpin it:
The errors ε₁, …, εₙ are unobserved random variables (also called noise).
The errors are independent of each other.
Each εᵢ follows a N(0, σ²) distribution. The constant variance σ² across all observations is called the homoskedasticity assumption.
β₀, β₁, and σ² are unknown but fixed parameters.
From these assumptions it follows that E(Yᵢ) = β₀ + β₁xᵢ and Var(Yᵢ) = σ². The expected value of Y at any given x is a linear function of x, and the spread of Y around that line is the same everywhere.
β₁ represents the change in the average response for a one-unit increase in x. This is the single most commonly tested and reported quantity in a regression analysis.
Example: if β₁ = 0.514 in the Galton height data, then for every additional inch of father's height, the son's height increases on average by about half an inch.
Given data (x₁, y₁), …, (xₙ, yₙ), the goal is to find values b₀ and b₁ that minimise:
f(b₀, b₁) = Σᵢ [yᵢ − (b₀ + b₁xᵢ)]²
The procedure: take partial derivatives of f with respect to b₀ and b₁, set both equal to zero, and solve.
Setting ∂f/∂b₀ = 0 gives:
b₀ = ȳ − b₁x̄
Setting ∂f/∂b₁ = 0 and substituting yields:
β̂₁ = Σᵢ(xᵢ − x̄)(yᵢ − ȳ) / Σᵢ(xᵢ − x̄)²
β̂₀ = ȳ − β̂₁x̄
These are the least squares estimates.
Using R, the command summary(lm(y ~ x)) produces the regression output. For Galton's 1,078 father–son pairs, the fitted line is:
ŷ = 33.887 + 0.514x
Interpretation: for the population from which these data were drawn, when father's height increases by 1 inch, the son's height increases on average by 0.514 inches. The slope is well below 1, illustrating regression to the mean.
Quantity | Formula |
|---|---|
Model | Yᵢ = β₀ + β₁xᵢ + εᵢ |
Slope estimate | β̂₁ = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)² |
Intercept estimate | β̂₀ = ȳ − β̂₁x̄ |
Expected response | E(Yᵢ) = β₀ + β₁xᵢ |
Variance of response | Var(Yᵢ) = σ² |
The simple linear regression model appears anywhere you want to quantify a straight-line relationship: predicting crop yield from rainfall, estimating house price from square footage, or relating dosage to therapeutic response. In engineering and the sciences, regression on experimental data is often the first step in confirming or refining a theoretical model.
Students often assume β₁ tells you about a causal effect. In an observational study (like Galton's), the slope measures association, not causation.
Students sometimes forget that Y is random but x is treated as fixed (or conditioned upon). The randomness in the model lives entirely in the error terms.
Confusing the true parameters (β₀, β₁) with their estimates (β̂₀, β̂₁) is common. The parameters are fixed unknowns; the estimates are computed from the data and vary from sample to sample.
Regression to the mean is not a bias in the data or the method. It is a genuine statistical phenomenon that arises whenever the correlation between x and y is less than perfect.
⚠️ Be able to write down the SLR model and state all four assumptions (unobserved errors, independence, normality with constant variance, fixed parameters).
⚠️ Know the closed-form formulas for β̂₁ and β̂₀ and be able to derive them by setting partial derivatives to zero.
⚠️ The interpretation of β̂₁ as "the change in the average response per one-unit increase in x" is a classic short-answer question.
⚠️ Understand why regression to the mean occurs and be able to explain it using the Galton example.
True or False: In the SLR model, both x and Y are random variables.
Fill in the blank: The assumption that Var(εᵢ) = σ² for all i is called the ________ assumption.
True or False: β̂₁ = 0.514 means every father who is one inch taller has a son who is exactly 0.514 inches taller.
Fill in the blank: The least squares estimates are found by minimising the sum of ________ residuals.
True or False: The regression line always passes through the point (x̄, ȳ).
Answers: 1. False (x is treated as fixed). 2. Homoskedasticity. 3. False (it describes the change in the average, not for every individual). 4. Squared. 5. True (this follows directly from β̂₀ = ȳ − β̂₁x̄).
Q: Write down the simple linear regression model and list its four key assumptions.
A: Yᵢ = β₀ + β₁xᵢ + εᵢ for i = 1, …, n. Assumptions: (1) ε₁, …, εₙ are unobserved random variables; (2) the errors are mutually independent; (3) each εᵢ ~ N(0, σ²) with constant variance (homoskedasticity); (4) β₀, β₁, and σ² are unknown, fixed parameters.
Q: Derive the least squares estimator for β₁.
A: Define f(b₀, b₁) = Σ[yᵢ − (b₀ + b₁xᵢ)]². Set ∂f/∂b₀ = 0 to obtain b₀ = ȳ − b₁x̄. Substitute into ∂f/∂b₁ = 0 and solve: β̂₁ = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)².
Q: In the Galton height data, the least squares line is ŷ = 33.887 + 0.514x. Interpret the slope.
A: For the population of father–son pairs from which the data were collected, each additional inch in the father's height is associated with an increase of approximately 0.514 inches in the son's height, on average.
Q: Explain the concept of regression to the mean using the Galton example.
A: You might expect that fathers of height x would have sons whose average height is about x + 1 (since sons were on average one inch taller). Instead, the sons' average height is pulled toward the overall mean. The slope of the line of averages is less than 1, meaning very tall fathers have sons who are tall but less extreme, and very short fathers have sons who are short but less extreme.
This material connects directly to multiple linear regression (Chapter 2 and beyond), where several predictors replace the single x. The ANOVA decomposition introduced in Part 2 of these notes reappears in every regression and analysis-of-variance context throughout the course. Correlation, covered at the end of this chapter, formalises the strength of the linear relationship that the slope captures.
simple linear regression, SLR, OLS, ordinary least squares, regression model, regression equation, slope, intercept, predictor variable, explanatory variable, covariate, regressor, feature, response variable, dependent variable, outcome variable, error term, noise, disturbance, homoskedasticity, constant variance, Galton, regression to the mean, reversion to the mean, line of best fit, fitted line, scatterplot, least squares line, beta hat, STAT 212, Principles of Statistics II