Source: Lecture Notes, Chapter 1 (Part C) – Dr. Abhishek Chakrabortty, Texas A&M University
Tags: model selection, AIC, BIC, R-squared, adjusted R-squared, residual analysis, standardised residuals, normal probability plot, influential observations, outliers, Cook's distance, Cook's D, leaps, all subsets regression, STAT 212
Difficulty: Intermediate Prerequisites: Parts 1 and 2 of these notes (Model Setup, Estimation, Inference, F-Test).
Fitting a multiple regression model is only half the job. You also need to decide which predictors to include and whether the model's assumptions hold. This section covers model selection (choosing among candidate models using R², AIC, and BIC), residual diagnostics (checking whether the assumptions of normality, constant variance, and independence are reasonable), and the detection of influential observations using Cook's distance. These tools help you move from "a model that runs" to "a model you can trust."
When comparing models with the same number of predictors, pick the one with the highest R². When models differ in complexity, use AIC or BIC (smaller is better). Residual plots check the model assumptions. Cook's distance flags observations that disproportionately influence the fitted coefficients.
Model selection
The process of choosing which predictors to include in the regression model. In simple terms, it is about finding a model that is complex enough to capture the real patterns but simple enough to avoid fitting noise.
AIC (Akaike Information Criterion)
A measure that balances model fit against complexity. Smaller AIC values indicate a better model. AIC penalises additional predictors less severely than BIC, so it tends to favour slightly more complex models.
BIC (Bayesian Information Criterion)
Similar to AIC but with a stronger penalty for the number of predictors. Smaller BIC values indicate a better model. BIC tends to favour simpler models than AIC, especially when n is large.
All subsets regression (leaps)
A procedure that fits every possible combination of predictors and reports the best model(s) at each model size, ranked by R². With k predictors, there are 2^k possible models. In R, the leaps command from the leaps package handles this.
Residual (eᵢ)
eᵢ = yᵢ − ŷᵢ. The difference between the observed and predicted value for observation i. If the model is correct, the residuals should behave like a random sample from a normal distribution.
Standardised residual (eᵢ)*
eᵢ* = (yᵢ − ŷᵢ) / σ̂. The residual divided by the estimated standard deviation of the errors. If the model is correct, standardised residuals should behave like draws from a standard normal distribution, N(0, 1).
Outlier
An observation that does not fit the pattern established by the rest of the data. Outliers come in two broad types, and they have different effects on the model.
Influential observation
An observation whose removal would substantially change the estimated coefficients or the estimated error variance.
Cook's distance (Cook's D)
A summary measure of how much all the fitted values change when observation i is removed from the dataset. It combines information about the size of the residual and the leverage (how unusual the predictor values are). In simple terms, a large Cook's D means that one data point is pulling the regression surface towards itself.
With k candidate predictors, there are 2^k possible models (including the intercept-only model). When k is small (5 or 6), fitting all of them and comparing is feasible.
Same number of predictors: Compare models using R². The model with the highest R² fits best.
Different numbers of predictors: Use AIC or BIC, because R² always increases (or stays the same) as you add more predictors, even if the new predictor is useless. AIC and BIC penalise model complexity to guard against overfitting.
Smaller AIC or BIC = better model
AIC tends to pick slightly larger models
BIC tends to pick slightly smaller models
Using leaps in R on the mesquite tree data:
No. vars | Best model | R² | AIC | BIC |
|---|---|---|---|---|
1 | ldiam1 | 0.818 | 26.06 | 29.04 |
2 | ldiam2, ltotht | 0.865 | 22.17 | 26.16 |
3 | ldiam1, ldiam2, ltotht | 0.878 | 22.04 | 27.02 |
4 | ldiam1, ldiam2, lcanht, ltotht | 0.884 | 23.07 | 29.05 |
AIC prefers the three-variable model (ldiam1, ldiam2, ltotht). BIC prefers the two-variable model (ldiam2, ltotht). Both agree that the full four-variable model is not the best choice, despite having the highest R².
If the model is correct, the residuals eᵢ = yᵢ − ŷᵢ should look like a random sample from a normal distribution. Standardised residuals eᵢ* = eᵢ / σ̂ should look like draws from N(0, 1).
Four standard diagnostic plots:
eᵢ vs. observation index i:* Checks independence of errors. Look for trends, cycles, or clustering over time. A random scatter is good.
eᵢ vs. fitted values ŷᵢ:* Checks constant variance (homoscedasticity). Look for a "funnel" or "fan" shape, which suggests the variance changes with the predicted value. A random horizontal band is good.
eᵢ vs. an independent variable:* Checks for nonlinear relationships the model has missed. A curve in this plot suggests a transformation or polynomial term may be needed.
Normal probability plot (QQ plot) or kernel density estimate of residuals: Checks normality. Points should fall roughly along a straight line in a QQ plot. Heavy tails, skewness, or outliers show up as deviations from the line.
There are two types of outliers, and they affect the model differently:
Type 1: The predictor values xᵢ are in the middle of the data, but yᵢ is unusually large or small. This kind of outlier inflates σ̂² but does not have much effect on the coefficient estimates β̂.
Type 2: The predictor values xᵢ are far from the rest of the data (high leverage). This kind of outlier can substantially shift the coefficient estimates β̂, pulling the regression surface towards itself.
In multiple regression, spotting outliers visually is difficult because you cannot plot Y against a high-dimensional predictor vector. This is where Cook's distance helps.
Let ŷ₁, ..., ŷₙ be the usual predicted values and let ŷ₁⁽ⁱ⁾, ..., ŷₙ⁽ⁱ⁾ be the predicted values when the model is refit after deleting observation i. Cook's D for observation i is:
Dᵢ = Σⱼ₌₁ⁿ (ŷⱼ − ŷⱼ⁽ⁱ⁾)² / (k · σ̂²)
One Cook's D value is computed for each of the n observations.
Interpretation guidelines:
D < 1: Generally not considered influential.
D > 1.5: Suggests a potentially influential observation. Refit the model without that case and examine how much the coefficient estimates change.
In R:
output = cooks.distance(fit)
plot(output, ylab="Cook's D")
This produces a plot of Cook's D values, making it easy to spot any that stand out.
Item | Formula / Command |
|---|---|
Standardised residual | eᵢ* = (yᵢ − ŷᵢ) / σ̂ |
Cook's D | Dᵢ = Σ(ŷⱼ − ŷⱼ⁽ⁱ⁾)² / (k · σ̂²) |
Number of possible models | 2^k |
R (all subsets) |
|
R (Cook's D) |
|
R (AIC) |
|
R (BIC) |
|
Model selection is a routine part of applied work. In epidemiology, researchers decide which risk factors to retain in a model predicting disease outcomes. In marketing, analysts choose which customer features best predict purchasing behaviour. Residual diagnostics are standard practice in any published regression analysis, because reviewers and readers want assurance that the model assumptions are not badly violated. Cook's distance is used in quality control (flagging unusual manufacturing runs) and in finance (identifying trades or data entries that distort a fitted model).
Students often think the model with the highest R² is always the best. R² never decreases when you add a predictor, even a completely irrelevant one. AIC and BIC exist precisely to penalise unnecessary complexity.
A common mistake is to inspect residual plots and declare "the residuals look random" without considering what specific patterns to look for (funnel shapes, trends, curvature, heavy tails).
Students sometimes believe that an outlier must always be removed. Outliers should be investigated, not automatically deleted. They may represent genuine extreme values, data entry errors, or a different population. Removing a valid observation because it is inconvenient biases the analysis.
Cook's D thresholds (1 or 1.5) are rules of thumb, not rigid cutoffs. A point with D = 0.9 might still deserve investigation, and a point with D = 1.6 might turn out to be legitimate after closer inspection.
⚠️ Know how to use AIC and BIC to compare models of different sizes, and which direction (smaller) is better.
⚠️ Be able to interpret a residual-vs-fitted-values plot and identify constant variance violations.
⚠️ Know the two types of outliers and how each affects the regression.
⚠️ Be able to state the formula for Cook's D and the interpretation guidelines (D < 1 generally fine, D > 1.5 worth investigating).
⚠️ Know the R commands: leaps for all subsets regression, cooks.distance(fit) for Cook's D.
⚠️ AIC vs. BIC: AIC favours slightly more complex models, BIC favours simpler ones. Be prepared to state which model each criterion selects.
True or False: R² always increases when you add another predictor to a model.
Fill in the blank: Smaller values of AIC or BIC indicate a _______ model.
True or False: An outlier with unusual predictor values (high leverage) primarily inflates σ̂² rather than shifting the coefficient estimates.
Fill in the blank: Cook's D values greater than _______ suggest a potentially influential observation.
True or False: Standardised residuals from a correctly specified model should behave like draws from N(0, 1).
Answers: 1. True (or at least stays the same). 2. Better. 3. False (high-leverage outliers shift the coefficient estimates; it is the other type, with unusual y but ordinary x, that mainly inflates σ̂²). 4. 1.5 (some sources use 1 as a softer threshold). 5. True.
Q: In the mesquite data, AIC selects the three-variable model (ldiam1, ldiam2, ltotht) while BIC selects the two-variable model (ldiam2, ltotht). Which model would you choose and why?
A: Either choice is defensible. BIC's two-variable model is simpler and requires only two measurements per tree, which is a practical advantage. AIC's three-variable model captures slightly more variance (R² = 0.878 vs. 0.865). If measurement cost matters, the BIC model is preferred. If predictive accuracy is the priority, the AIC model is a reasonable choice. In the lecture, the emphasis was on the value of simplicity.
Q: You plot standardised residuals against fitted values and see a clear funnel shape (residuals spread out as ŷ increases). What assumption is violated and what might you do?
A: The constant variance (homoscedasticity) assumption is violated. Possible remedies include applying a variance-stabilising transformation to Y (such as log or square root), or using weighted least squares.
Q: Observation 2 in the mesquite dataset is a large tree (diam1 = 5.2, diam2 = 4.0, leafwt = 4052). If its Cook's D value were 2.1, what would you do?
A: A Cook's D of 2.1 exceeds the 1.5 guideline, so this observation is potentially influential. The next step is to refit the model without observation 2 and compare the new coefficient estimates to the originals. If they change substantially, report both sets of results and investigate whether the observation is a valid extreme case or a data error.
Q: Why can you not simply use R² to choose between a model with 2 predictors and a model with 4 predictors?
A: R² always increases (or stays the same) as predictors are added, even if the additional predictors have no real relationship with Y. Comparing models of different sizes by R² alone will always favour the larger model. AIC and BIC correct for this by penalising the number of parameters.
Q: State the R commands to (a) fit all possible subsets of a regression and (b) compute Cook's distance for a fitted model fit.
A: (a) Use the leaps function from the leaps package. (b) cooks.distance(fit).
Model selection connects to the bias-variance trade-off discussed in more advanced statistics and machine learning courses: simpler models have more bias but less variance in their predictions, and the goal is to find the sweet spot. Residual diagnostics connect back to the assumptions introduced in simple regression and carry forward into generalised linear models, time series, and any regression-based method. Cook's distance is one of a family of influence diagnostics (others include DFBETAS and leverage values) that become increasingly important in high-dimensional settings.
Related Terms / Search Tags: model selection, AIC, BIC, Akaike information criterion, Bayesian information criterion, all subsets regression, leaps, R-squared, adjusted R-squared, residual plots, standardised residuals, normal probability plot, QQ plot, homoscedasticity, constant variance, outliers, influential observations, leverage, Cook's distance, Cook's D, diagnostic plots, STAT 212, mesquite tree data, overfitting, model complexity