Difficulty: Intermediate | Prerequisites: Part 1 of these notes (SLR model, least squares estimation, residuals).
Fitting a regression line is only half the job. The other half is checking whether the model's assumptions hold and measuring how well the line explains the data. Residual plots are the primary diagnostic tool, and R² is the primary summary measure. This section bridges the gap between "I have a fitted line" and "I trust (or do not trust) this fitted line." You need Part 1's definitions of residuals, SSE, and the four model assumptions before this will make sense.
Residuals are your window into whether the SLR assumptions hold. Plot them several ways to check independence, linearity, constant variance, and normality. R² (the coefficient of determination) tells you what fraction of the total variation in Y is explained by the linear relationship with x.
Residual (e_i)
e_i = Y_i − Ŷ_i. The observed difference between the actual response and the value predicted by the fitted line. Think of it as the bit the model could not explain for observation i.
Total sum of squares (SST)
SST = Σ(Y_i − Ȳ)². Measures all the variation in the response values, from whatever source. In simple terms, this is the total spread of Y around its own mean.
Regression sum of squares (SSR)
SSR = Σ(Ŷ_i − Ȳ)². The portion of total variation in Y that the linear model accounts for. In simple terms, this is the "explained" part.
ANOVA decomposition
SST = SSR + SSE. Total variation splits cleanly into the part explained by the regression and the part left over as error. Think of it as a budget: everything in SST is allocated to either SSR or SSE.
Coefficient of determination (R²)
R² = SSR / SST = 1 − SSE / SST. The fraction of total variation in Y explained by the linear relationship with x. Always between 0 and 1.
Durbin–Watson statistic
A test statistic for detecting autocorrelation (dependence) in residuals. It should be close to 2 when errors are independent with zero mean and constant variance. In simple terms, if consecutive residuals tend to be similar to each other, the Durbin–Watson statistic drifts away from 2.
Heteroscedasticity
The condition where the variance of the errors changes across values of x or Ŷ. The opposite of homoscedasticity. A "wedge-shaped" residual plot is the classic visual signal.
Normal Q–Q plot
A plot that compares the quantiles of the residuals to the quantiles of a theoretical normal distribution. If the residuals are approximately normal, the points line up along a straight diagonal.
If the SLR model is correct, the residuals e₁, …, e_n should behave roughly like a random sample from a N(0, σ²) distribution. Any systematic pattern in the residuals signals a violated assumption.
Residuals vs. observation order (e_i vs. i)
Purpose: checks for dependence due to the sequencing of observations.
What to look for: a random scatter with no trend or cyclical pattern.
Formal companion: the Durbin–Watson statistic, which should sit near 2.
Residuals vs. predictor (e_i vs. x_i)
Purpose: checks whether the true relationship is more complex than a straight line, and whether the zero-mean assumption holds.
What to look for: a "flat" band of points centred on zero. A curved pattern suggests a non-linear relationship.
Residuals vs. fitted values (e_i vs. Ŷ_i)
Purpose: checks the equal-variance (homoscedasticity) assumption.
What to look for: constant vertical spread across the plot. A "wedge shape," where the spread fans out (or narrows), signals heteroscedasticity.
Histogram, density plot, or normal Q–Q plot of residuals
Purpose: checks the normality assumption.
What to look for: a roughly bell-shaped histogram, or Q–Q points falling close to the y = x line.
The lecture notes show residual plots for Galton's father–son height data.
The residuals-vs-x plot looks flat, supporting the linearity and zero-mean assumptions.
The residuals-vs-Ŷ plot also looks flat, supporting constant variance.
The histogram is approximately bell-shaped and the Q–Q plot is nearly linear, supporting normality.
All four assumptions appear reasonable for this dataset.
A curved residual-vs-x pattern (the notes show one with a clear U-shape) means a straight line is the wrong model; the true relationship is non-linear.
A wedge-shaped residual-vs-Ŷ plot means the variance grows (or shrinks) with Ŷ, violating homoscedasticity. Transformations of Y (e.g. log) or weighted least squares are typical remedies.
The algebraic identity is:
SST = SSR + SSE
The cross-product term Σ(Y_i − Ŷ_i)(Ŷ_i − Ȳ) equals zero. The proof uses the facts that Ŷ_i − Ȳ = β̂₁(x_i − x̄) and Y_i − Ŷ_i = Y_i − Ȳ − β̂₁(x_i − x̄), then shows the resulting sum telescopes to zero.
R² = SSR / SST = 1 − SSE / SST.
R² = 1 means the line explains all the variation (every point sits on the line).
R² = 0 means the line explains none of the variation (knowing x does not help predict Y at all).
For Galton's data, R² = 0.251, so roughly 25% of the variation in sons' heights is explained by the linear relationship with fathers' heights. The remaining 75% comes from other sources (genetics from the mother, environment, nutrition, etc.).
The lecture notes include scatterplots at several R² values:
R² ≈ 0.006: essentially a cloud with no discernible trend.
R² ≈ 0.175: a vague trend is visible, but the scatter is heavy.
R² ≈ 0.491: a clear trend, but individual predictions remain imprecise.
R² ≈ 0.908: a tight linear band; the model captures most of the variation.
fit <- lm(y ~ x)
resid <- fit$residuals
predict <- fit$fitted.values
# Diagnostic plots
plot(x, resid, xlab = "x", ylab = "Residuals")
plot(predict, resid, xlab = "yhat", ylab = "Residuals")
hist(resid)
plot(density(resid))
qqnorm(resid)
SST = Σ(Y_i − Ȳ)²
SSR = Σ(Ŷ_i − Ȳ)²
SSE = Σ(Y_i − Ŷ_i)² = Σ e_i²
ANOVA identity: SST = SSR + SSE
R² = SSR / SST = 1 − SSE / SST
Durbin–Watson = Σ(e_{i+1} − e_i)² / Σ e_i² (sum from i = 1 to n−1 in the numerator, 1 to n in the denominator)
Residual diagnostics are how engineers, scientists, and analysts decide whether their model is trustworthy before making forecasts or decisions. In quality control, a wedge-shaped residual plot might reveal that measurement precision degrades for larger items, prompting a redesigned process. In finance, autocorrelated residuals might signal that a time-series model is missing a trend.
Students often think a high R² means the model is "correct." It does not. R² can be high even when the linearity assumption is badly violated (e.g. fitting a line to a strong but curved relationship). Always check the residual plots.
A low R² does not mean the model is useless. Even an R² of 0.25 (like Galton's data) confirms a real, statistically significant relationship; it simply means other factors also matter.
Students sometimes believe the ANOVA decomposition SST = SSR + SSE requires special assumptions. It is a purely algebraic identity that holds for any least-squares fit.
Confusing the Durbin–Watson statistic's "close to 2" benchmark with "equal to 2 is perfect" is common. Values near 2 are fine; values near 0 or near 4 indicate positive or negative autocorrelation, respectively.
⚠️ Be able to state what each of the four diagnostic plots checks and what a "good" version looks like.
⚠️ Know the ANOVA decomposition SST = SSR + SSE and how R² is derived from it.
⚠️ Be able to interpret R² in context: "approximately __% of the variation in Y is explained by the linear relationship with x."
⚠️ Recognise a wedge-shaped residual plot as evidence of heteroscedasticity and a curved residual plot as evidence of non-linearity.
Fill in the blank: SST = ____ + ____. (SSR + SSE)
True or false: R² = 0.90 guarantees that the linear model is appropriate. (False – the residual plots could still reveal assumption violations.)
True or false: The Durbin–Watson statistic should be close to 0 when errors are independent. (False – it should be close to 2.)
Fill in the blank: A wedge-shaped residual-vs-Ŷ plot signals ____. (Heteroscedasticity, i.e. non-constant variance.)
True or false: R² for Galton's father–son data is approximately 0.25. (True.)
Q: Define SST, SSR, and SSE, and state the relationship among them.
A: SST = Σ(Y_i − Ȳ)² is the total variation. SSR = Σ(Ŷ_i − Ȳ)² is the variation explained by regression. SSE = Σe_i² is the unexplained (residual) variation. They satisfy SST = SSR + SSE.
Q: For a dataset with R² = 0.60, what does this tell you?
A: Approximately 60% of the variation in the response variable is explained by the linear relationship with the predictor. The remaining 40% is due to other factors or random noise.
Q: Describe what you would look for in a residual-vs-fitted-values plot to check the constant-variance assumption.
A: The vertical spread of the residuals should remain roughly the same across all fitted values. If the spread fans out (or narrows) as Ŷ increases, the equal-variance assumption is violated (heteroscedasticity).
Q: A residual plot of e_i vs. x_i shows a clear U-shaped curve. What does this indicate?
A: The true relationship between Y and x is non-linear. A straight-line model is inadequate, and a higher-order term (e.g. x²) or a transformation may be needed.
Q: What R command produces a normal Q–Q plot of the residuals stored in a vector called resid?
A: qqnorm(resid)
R² reappears in multiple regression, where it is adjusted for the number of predictors (adjusted R²). Residual diagnostics carry over to every regression model in the course. The ANOVA decomposition here is a special case of the broader ANOVA framework you will study later. The correlation coefficient ρ̂ (covered in Part 3 of these notes) satisfies ρ̂² = R² in simple linear regression.
residual, residual plot, diagnostics, SST, SSR, SSE, ANOVA decomposition, R-squared, coefficient of determination, Durbin–Watson, autocorrelation, homoscedasticity, heteroscedasticity, constant variance, Q–Q plot, normality check, STAT 212, Principles of Statistics II, Texas A&M