Source: STAT 212 Lecture Notes, Texas A&M University Difficulty: Intermediate | Prerequisites: Polynomial regression foundations (Part 1 of these notes), simple linear regression inference
Tags: adjusted R-squared, AIC, BIC, model selection, overfitting, polynomial degree, confidence interval, prediction interval, hypothesis testing, t-distribution, STAT 212
Once you know how to fit a polynomial, the next question is: which degree should you choose? A higher-degree polynomial always fits the training data better (R² goes up), but it may be capturing noise rather than real structure. This section introduces three model selection criteria, adjusted R², AIC, and BIC, that penalise model complexity to guide you toward the right degree. It also covers inference for polynomial regression: confidence intervals and hypothesis tests for individual coefficients, confidence intervals for the mean response, and prediction intervals for a new observation. All the R commands from simple linear regression carry over unchanged.
Plain R² always increases with polynomial degree, so it cannot be used to choose the right model. Adjusted R², AIC, and BIC all add a complexity penalty. BIC penalises most heavily and tends to select simpler models. For inference, the t-distribution with n − k − 1 degrees of freedom is used for coefficient tests and intervals, and the formulas for confidence and prediction intervals generalise from the straight-line case using the design matrix.
R² (coefficient of determination)
R² = SSR / SST = 1 − SSE / SST. It measures the fraction of total variation in the yᵢ values explained by the fitted model. Think of it as "how much of the scatter in Y does the model account for?"
Overfitting
Fitting a model that is too complex for the data, so it captures noise as well as signal. The model performs well on the training data but poorly on new data.
Adjusted R² (R²_adj)
R²_adj,k = [(n − 1)R²_k − k] / (n − 1 − k), or equivalently R²_k − [k / (n − 1 − k)] × (1 − R²_k). It penalises R² for adding more parameters. In simple terms, it asks "is the improvement in fit worth the extra complexity?"
AIC (Akaike Information Criterion)
AIC_k = n log(SSE_k) + 2k. A model selection criterion that balances fit (through SSE) against complexity (through 2k). Lower values are better. Choose the polynomial degree that minimises AIC.
BIC (Bayesian Information Criterion)
BIC_k = n log(SSE_k) + k log(n). Similar to AIC but with a heavier penalty for complexity because log(n) > 2 for any sample size n ≥ 8. BIC tends to select simpler models than AIC. Lower values are better.
Bootstrap (for polynomial regression)
A resampling method used to assess the variability of a fitted polynomial curve when only one dataset is available. Covered in the supplementary notes.
Standard error of β̂ᵢ
σ√cᵢ₊₁, where cⱼ is the jth diagonal element of (XᵀX)⁻¹. In practice, σ is replaced by σ̂ = √MSE.
Confidence interval for the mean response
An interval for the average value of Y at a given x₀. It uses the standard error of the fitted value μ̂(x₀).
Prediction interval
An interval for a single new observation of Y at x₀. It is wider than the confidence interval because it accounts for both estimation uncertainty and the random error ε of the new observation.
R²_k is guaranteed to increase (or at least not decrease) as the polynomial degree k increases.
Specifically: R²₁ < R²₂ < ⋯ < R²_{n−2} < R²_{n−1} = 1.
A degree n − 1 polynomial passes through every data point, giving R² = 1, but this is pure overfitting with no predictive value.
Plain R² cannot distinguish between meaningful improvement in fit and overfitting. You need a criterion that penalises complexity.
R²_adj,k = R²_k − [k / (n − 1 − k)] × (1 − R²_k).
The penalty term [k / (n − 1 − k)] × (1 − R²_k) grows with k, so adding a parameter that does not meaningfully reduce SSE will cause R²_adj to decrease.
Choose the degree k that maximises R²_adj.
In the lecture simulation (cubic data, n = 30), R²_adj was maximised at degree 3, correctly identifying the true model.
AIC_k = n log(SSE_k) + 2k. The penalty for each additional parameter is 2.
BIC_k = n log(SSE_k) + k log(n). The penalty for each additional parameter is log(n), which exceeds 2 whenever n ≥ 8 (i.e., almost always in practice).
Both are minimised to select the best model. Lower is better.
BIC penalises complexity more heavily, so it tends to select simpler (lower-degree) models than AIC.
In the lecture simulation with 1000 datasets generated from a cubic model (n = 30), the percentage of times each method selected the true degree (k = 3) was: BIC 84.0%, AIC 52.5%, adjusted R² 23.1%.
BIC was the clear winner. AIC was second. Adjusted R² had a strong tendency to overestimate the polynomial degree, selecting k = 12 in 17.2% of cases.
All three methods improve with larger sample sizes.
Fit the model: fit = lm(y ~ poly(x, 3, raw = T))
Get AIC: AIC(fit)
Get BIC: BIC(fit)
Get adjusted R²: summary(fit)$adj
In practice, fit polynomials of degree 1 through some reasonable maximum, compute these criteria for each, and select the degree that minimises AIC or BIC (or maximises adjusted R²).
The standard error of β̂ᵢ is σ√cᵢ₊₁, where cⱼ is the jth diagonal element of (XᵀX)⁻¹.
Under the normality assumption, (β̂ᵢ − βᵢ) / (σ̂√cᵢ₊₁) follows a t-distribution with n − k − 1 degrees of freedom.
A (1 − α) × 100% confidence interval for βᵢ is: β̂ᵢ ± t_{n−k−1, α/2} × σ̂√cᵢ₊₁.
To test H₀: βₖ = 0 vs. Hₐ: βₖ ≠ 0 is to ask whether the highest-degree term is needed. If H₀ is not rejected, a lower-degree polynomial may suffice.
Define x = (1, x₀, x₀², …, x₀ᵏ)ᵀ.
Point estimate of the mean response: μ̂(x₀) = β̂₀ + β̂₁x₀ + ⋯ + β̂ₖx₀ᵏ.
Confidence interval for the mean of Y at x₀: μ̂(x₀) ± t_{n−k−1, α/2} × σ̂ × √(xᵀ(XᵀX)⁻¹x).
Prediction interval for a new Y at x₀: μ̂(x₀) ± t_{n−k−1, α/2} × σ̂ × √(1 + xᵀ(XᵀX)⁻¹x).
The prediction interval is always wider because of the extra "1" under the square root, which accounts for the variability of the individual observation around its mean.
All commands from simple linear regression apply directly: summary(fit), confint(fit), predict(fit, newdata, interval = "confidence"), predict(fit, newdata, interval = "prediction").
Two equivalent ways to fit a third-degree polynomial: lm(y ~ poly(x, 3, raw = T)) or lm(y ~ x + I(x^2) + I(x^3)). Both produce the same predicted values and residuals.
Adjusted R²: R²_adj,k = [(n − 1)R²_k − k] / (n − 1 − k) = R²_k − [k / (n − 1 − k)](1 − R²_k)
AIC: AIC_k = n log(SSE_k) + 2k
BIC: BIC_k = n log(SSE_k) + k log(n)
Standard error of β̂ᵢ: SE(β̂ᵢ) = σ̂√cᵢ₊₁, where cⱼ = jth diagonal of (XᵀX)⁻¹
Confidence interval for βᵢ: β̂ᵢ ± t_{n−k−1, α/2} × σ̂√cᵢ₊₁
Confidence interval for mean response at x₀: μ̂(x₀) ± t_{n−k−1, α/2} × σ̂√(xᵀ(XᵀX)⁻¹x)
Prediction interval for Y at x₀: μ̂(x₀) ± t_{n−k−1, α/2} × σ̂√(1 + xᵀ(XᵀX)⁻¹x)
Model selection comes up every time you build a regression model in practice. In pharmaceutical dose-response studies, for instance, researchers fit polynomials to describe how a drug's effect varies with dose and use AIC or BIC to determine whether a quadratic or cubic term is justified by the data. In climate science, polynomial trends are fitted to temperature records, and choosing the wrong degree can lead to misleading extrapolations.
Students often think a higher R² always means a better model. It does not. R² is guaranteed to increase with polynomial degree, so it tells you nothing about whether additional terms are capturing signal or noise. Use adjusted R², AIC, or BIC instead.
A common error is to confuse the confidence interval for the mean response with the prediction interval. The confidence interval is for the average Y at x₀ across many hypothetical repetitions. The prediction interval is for a single new Y. The prediction interval is always wider.
Some students assume BIC is always "better" than AIC because it performed best in the lecture simulation. The comparison depends on the true model and sample size. BIC is more conservative (favours simpler models), which is advantageous when the true model is indeed simple.
When testing H₀: β₃ = 0 in a cubic model, students sometimes forget that failing to reject does not mean the quadratic model is "true." It means you lack sufficient evidence to conclude the cubic term is needed.
⚠️ The formula for adjusted R² is frequently tested. Make sure you can compute it given R²_k, n, and k.
⚠️ Know the direction of comparison: maximise adjusted R², but minimise AIC and BIC. Getting these backwards is an easy mark to lose.
⚠️ BIC penalises more than AIC (because log(n) > 2 for n ≥ 8). This is a standard short-answer question.
⚠️ The degrees of freedom for t-tests and confidence intervals in polynomial regression are n − k − 1, not n − 2. Make sure you use the right value.
⚠️ The difference between confidence intervals and prediction intervals (the extra "1" under the root) is a favourite exam distinction.
⚠️ Know the R commands: AIC(fit), BIC(fit), summary(fit)$adj, and the two equivalent ways to specify a polynomial in lm().
True or false: BIC will always select a polynomial degree less than or equal to that selected by AIC.
A: False. BIC tends to select simpler models but does not always do so for every dataset.
Fill in the blank: The penalty term in AIC for each additional parameter is ____, while in BIC it is ____.
A: 2; log(n).
True or false: A prediction interval for a single new observation is narrower than a confidence interval for the mean response at the same x₀.
A: False. The prediction interval is always wider.
Fill in the blank: To test whether a cubic term is needed in a cubic polynomial model, you test H₀: ____ = 0.
A: β₃.
Q: Explain why maximising R² is unreliable for selecting polynomial degree.
A: R² is guaranteed to be a non-decreasing function of polynomial degree. It increases every time you add a higher-order term, regardless of whether that term captures real structure or just noise. An (n − 1)-degree polynomial achieves R² = 1 by passing through every data point, which is pure overfitting with no generalisation ability.
Q: A researcher fits polynomials of degree 1 through 8 to a dataset with n = 40. The AIC values are 120, 98, 85, 84, 85, 87, 89, 92. Which degree should they select, and why?
A: Degree 4, because it has the lowest AIC (84). AIC is minimised at degree 4, suggesting the improvement from degree 3 to 4 is worth the extra parameter, but going beyond degree 4 adds complexity without improving fit enough to offset the penalty.
Q: In the lecture simulation, BIC selected the correct degree (3) in 84% of 1000 datasets, while adjusted R² selected it only 23.1% of the time. What was adjusted R² doing wrong?
A: Adjusted R² was overestimating the degree. It selected degree 12 in 17.2% of cases and spread its selections across higher degrees. Its penalty for complexity is milder than BIC's, so it is more prone to choosing unnecessarily complex models, especially with moderate sample sizes.
Q: Write the prediction interval formula for Y at x₀ in a polynomial regression of degree k, and explain what makes it wider than the confidence interval for the mean.
A: The prediction interval is μ̂(x₀) ± t_{n−k−1, α/2} × σ̂ × √(1 + xᵀ(XᵀX)⁻¹x). Compared to the confidence interval formula, which has √(xᵀ(XᵀX)⁻¹x) without the "1 +", the prediction interval includes the additional variance σ² of the individual observation's error term ε. That extra source of uncertainty makes the interval wider.
Q: In R, give two equivalent ways to fit a quadratic polynomial to data stored in vectors x and y.
A: lm(y ~ poly(x, 2, raw = T)) and lm(y ~ x + I(x^2)). Both produce the same fitted values and residuals.
Model selection criteria (AIC, BIC, adjusted R²) reappear throughout STAT 212 and in more advanced courses whenever you compare models of differing complexity, including multiple regression with different subsets of predictors.
The confidence interval vs. prediction interval distinction also carries over directly to multiple regression and is a core concept in any applied statistics course.
The hypothesis test H₀: βₖ = 0 is a special case of the general framework of testing whether a predictor is "significant," which becomes central in multiple regression.
adjusted R-squared, R-squared, AIC, Akaike information criterion, BIC, Bayesian information criterion, model selection, polynomial degree selection, overfitting, underfitting, confidence interval, prediction interval, mean response, t-distribution, degrees of freedom, hypothesis testing, R commands, poly function, lm function, STAT 212, Texas A&M