Source: Principles of Statistics II, Texas A&M University
Tags: AIC, BIC, model selection, hypothesis testing, t-test regression, confidence interval regression, prediction interval, nonlinear least squares, log transformation, variable transformation, heteroscedasticity, R programming regression, poly function R
Difficulty: Intermediate Prerequisites: Part 1 of these notes (polynomial regression model, least squares estimation, R² and adjusted R²). Familiarity with t-distributions and confidence intervals from introductory statistics.
Fitting a polynomial model is only the first step. You then need to decide which degree of polynomial to keep (model selection), test whether individual terms contribute meaningfully (inference), and quantify the uncertainty around predictions (intervals). Beyond polynomials, some relationships follow known functional forms such as exponential or power curves, and variable transformations can convert these into linear models that are easier to estimate. This second set of notes covers those tools and connects them to the estimation foundations from Part 1.
AIC and BIC provide formal criteria for choosing polynomial degree by penalising complexity, with BIC favouring simpler models more strongly. Coefficient inference uses t-tests built from the residual variance and the design matrix. Confidence intervals estimate the mean response at a given x, while prediction intervals (always wider) cover a single future observation. When the true relationship is exponential or power-law rather than polynomial, log and log-log transformations can linearise the model and stabilise variance.
Akaike Information Criterion (AIC)
AIC = n·log(SSE/n) + 2k. A model-selection metric that balances goodness of fit (via SSE) against the number of parameters k. Lower AIC is better.
Think of it as a score that rewards a good fit but charges a fee for every parameter you add.
Bayesian Information Criterion (BIC)
BIC = n·log(SSE/n) + k·log(n). Similar to AIC but with a heavier penalty for additional parameters, especially as the sample size n grows. Lower BIC is better.
In simple terms, BIC is the stricter sibling of AIC: it punishes complexity more and tends to select simpler models.
Residual variance (σ̂²)
The estimated variance of the error term, calculated as σ̂² = SSE / (n – k – 1). It is the average squared residual, adjusted for degrees of freedom, and feeds into standard errors, t-tests, and intervals.
Think of it as a measure of how much scatter remains after the model has done its best.
Standard error of a coefficient, SE(β̂ⱼ)
Derived from the diagonal elements of σ̂²(X⊤X)⁻¹. It quantifies the uncertainty in each estimated coefficient and is used to build confidence intervals and t-statistics.
In simple terms, it tells you how precisely you have estimated a given β.
t-statistic (for regression coefficients)
t = β̂ⱼ / SE(β̂ⱼ), compared against a t-distribution with n – k – 1 degrees of freedom. Used to test H₀: βⱼ = 0 (i.e., whether the j-th term contributes to the model).
Think of it as the signal-to-noise ratio for a single coefficient.
Confidence interval for the mean response
An interval estimate for the expected (average) value of Y at a particular x₀. It accounts for uncertainty in the estimated regression function but not for the variability of individual observations.
Prediction interval for a future observation
An interval estimate for a single new observation at x₀. It is always wider than the confidence interval because it includes both the uncertainty in the mean and the inherent variability of individual data points.
Nonlinear least squares
An estimation method for models where the parameters enter the function in a nonlinear way (e.g., Y = αe^(βx) + ε). Unlike polynomial regression, the normal equations do not yield a closed-form solution, so iterative numerical algorithms are required.
In simple terms, it is the "trial and improvement" version of least squares, for models where the algebra cannot be solved in one step.
Logarithmic transformation
Taking the natural log of the response (and sometimes the predictor) to convert a nonlinear model into a linear one. For an exponential model Y = αe^(βx)ε, logging both sides gives log Y = log α + βx + log ε, which is linear in its parameters.
Log-log transformation
Logging both Y and x to linearise a power-law model. For Y = αx^β ε, the transformed model is log Y = log α + β log x + log ε. The slope β in the log-log model is the elasticity: the percentage change in Y per one-percent change in x.
Heteroscedasticity
Non-constant variance of the residuals. When the spread of residuals increases (or decreases) with x or ŷ, the assumption of equal variance is violated. Transformations such as log(Y) can often stabilise this.
In simple terms, if the residual plot fans out like a trumpet, you have heteroscedasticity.
After fitting several candidate polynomial models (degree 1, 2, 3, ...), you need a principled way to choose among them.
AIC = n·log(SSE/n) + 2k
BIC = n·log(SSE/n) + k·log(n)
Both reward lower SSE (better fit) and penalise higher k (more parameters). The key difference is the penalty term:
AIC adds 2 per parameter, regardless of sample size.
BIC adds log(n) per parameter, which grows with sample size. For any n ≥ 8, BIC's penalty exceeds AIC's, so BIC tends to prefer simpler models.
In practice, compute AIC and BIC for each candidate degree and select the model with the lowest value. When AIC and BIC disagree, consider the context: BIC is more conservative and may be preferred when interpretability matters.
Once you have fitted a polynomial, you can test whether each term is needed.
The residual variance is σ̂² = SSE / (n – k – 1).
Standard errors come from the diagonal of σ̂²(X⊤X)⁻¹.
The t-statistic for testing H₀: βⱼ = 0 is t = β̂ⱼ / SE(β̂ⱼ), with n – k – 1 degrees of freedom.
A confidence interval for βⱼ is β̂ⱼ ± t(n–k–1, α/2) × SE(β̂ⱼ).
If the t-test for the highest-degree term is not significant, that term may be dropped, and the model can be refitted with a lower degree.
Both intervals are centred on ŷ(x₀), but they answer different questions.
Confidence interval for the mean response at x₀:
ŷ(x₀) ± t(n–k–1, α/2) × σ̂ × √(x₀⊤(X⊤X)⁻¹x₀)
where x₀ = (1, x₀, x₀², ..., x₀ᵏ)⊤.
This interval captures where the true average of Y sits at this x value. It is narrower because it only reflects estimation uncertainty.
Prediction interval for a new observation at x₀:
ŷ(x₀) ± t(n–k–1, α/2) × σ̂ × √(1 + x₀⊤(X⊤X)⁻¹x₀)
The extra "1" under the square root accounts for the variability of a single observation around the mean. Prediction intervals are always wider than confidence intervals for the same x₀ and confidence level.
Both intervals are narrowest near the centre of the data and widen as x₀ moves towards the edges or beyond the observed range.
Extrapolating far beyond the data range produces very wide (and unreliable) intervals.
Two equivalent approaches:
Using poly():
fit <- lm(y ~ poly(x, degree, raw = TRUE))
Using explicit terms with I():
fit <- lm(y ~ x + I(x^2) + I(x^3))
summary(fit) returns coefficients, standard errors, t-statistics, p-values, R², and adjusted R².
confint(fit) gives confidence intervals for each coefficient.
predict(fit, newdata = ..., interval = "confidence") or interval = "prediction" produces the relevant intervals for new x values.
AIC(fit) and BIC(fit) return the information criteria directly.
When the underlying relationship follows a known functional form such as exponential growth, logistic growth, or a trigonometric pattern, the model is written as:
Y = f(x; θ) + ε
where θ is a vector of parameters that enter f in a nonlinear way.
Estimation uses nonlinear least squares, which minimises Σ(yᵢ – f(xᵢ; θ))² iteratively.
There is no closed-form solution; algorithms such as Gauss-Newton or Levenberg-Marquardt refine initial guesses until convergence.
Good starting values matter: poor initial guesses can lead to convergence failures or local minima.
In R, nls() handles nonlinear least squares.
Rather than using iterative methods, you can sometimes transform the data so that an ordinary linear regression applies.
Exponential model: Y = αe^(βx)ε
Take logs: log Y = log α + βx + log ε
This is linear in (log α) and β, so fit a simple linear regression of log Y on x.
Power-law model: Y = αx^β ε
Take logs of both sides: log Y = log α + β log x + log ε
Fit a linear regression of log Y on log x. The slope β is the elasticity.
Why transformations also help with variance:
When the spread of Y increases with x (heteroscedasticity), taking log Y often compresses the larger values more than the smaller ones, stabilising the variance.
After transforming, check the residual plot again to confirm the variance looks more constant.
Historical wheat prices tend to grow exponentially over centuries. Fitting a linear model to the raw data produces residuals that fan out and show curvature. Taking the log of the price index linearises the trend, and the residuals from the log-scale model show much less pattern and more constant spread. This is a textbook illustration of both linearisation and variance stabilisation in one step.
AIC: AIC = n·log(SSE/n) + 2k
BIC: BIC = n·log(SSE/n) + k·log(n)
Residual variance: σ̂² = SSE / (n – k – 1)
Confidence interval for βⱼ: β̂ⱼ ± t(n–k–1, α/2) × SE(β̂ⱼ)
Confidence interval for mean response at x₀: ŷ(x₀) ± t(n–k–1, α/2) × σ̂ × √(x₀⊤(X⊤X)⁻¹x₀)
Prediction interval at x₀: ŷ(x₀) ± t(n–k–1, α/2) × σ̂ × √(1 + x₀⊤(X⊤X)⁻¹x₀)
Log transformation (exponential model): log Y = log α + βx + log ε
Log-log transformation (power-law model): log Y = log α + β log x + log ε
AIC and BIC are the standard model-selection tools across the sciences, from ecology (choosing among species-abundance models) to finance (selecting lag orders in time-series models). Prediction intervals matter whenever a single future outcome needs bounding, for instance, in quality control (will the next manufactured part be within tolerance?) or weather forecasting. Log transformations are routine in economics and epidemiology, where data such as income, prices, and viral counts naturally span several orders of magnitude and exhibit multiplicative rather than additive error.
Students often confuse confidence intervals and prediction intervals. The confidence interval is for the mean of Y at x₀; the prediction interval is for a single new Y. The prediction interval is always wider. Exams frequently test this distinction.
A common error is interpreting AIC or BIC values in absolute terms. The numbers themselves mean nothing in isolation; only differences between models matter. The model with the lowest AIC (or BIC) wins.
Students sometimes apply a log transformation and then interpret the coefficients as if they were on the original scale. After a log transformation, the coefficients describe relationships on the log scale; back-transforming to the original scale requires exponentiation, and the interpretation changes (e.g., multiplicative rather than additive effects).
When using poly(x, degree) in R without raw = TRUE, the function produces orthogonal polynomials, whose coefficients are not directly comparable to the raw-coefficient form. This catches students who forget the raw = TRUE argument and then try to interpret β₂ as the coefficient on x².
⚠️ Be able to state the AIC and BIC formulas and explain why BIC penalises complexity more heavily for large n.
⚠️ Know how to construct a confidence interval for a coefficient using the t-distribution and the standard error from (X⊤X)⁻¹.
⚠️ Expect a question requiring you to distinguish between a confidence interval for the mean response and a prediction interval. Know that the prediction interval has the extra "+1" under the root.
⚠️ Be prepared to take a nonlinear model (exponential or power-law), apply a log transformation, and identify the resulting linear model's parameters.
⚠️ R syntax for lm(), poly(), I(), summary(), confint(), predict(), AIC(), and BIC() is fair game.
True or false: BIC always selects a model with more parameters than AIC.
A: False. BIC has a heavier penalty for complexity (for n ≥ 8) and therefore tends to select simpler models than AIC.
Fill in the blank: The prediction interval is wider than the confidence interval because it accounts for _______ in addition to estimation uncertainty.
A: The variability (randomness) of a single new observation around the mean.
True or false: After fitting log Y = a + b·log x, the coefficient b represents the additive change in Y per unit increase in x.
A: False. In a log-log model, b is the elasticity: the approximate percentage change in Y for a one-percent change in x.
Fill in the blank: In R, to fit a raw polynomial of degree 4, you write lm(y ~ poly(x, ____, raw = ____)).
A: poly(x, 4, raw = TRUE).
Q: You fit polynomial models of degree 1 through 5 and obtain the following BIC values: 210, 195, 192, 194, 198. Which model do you select, and why?
A: Select the degree-3 model (BIC = 192) because it has the lowest BIC. The degree-2 model is close (195), but the degree-3 model achieves a better balance of fit and parsimony according to this criterion.
Q: Explain the difference between a confidence interval and a prediction interval at x₀ = 5, and state which is wider.
A: The confidence interval estimates where the true mean of Y lies when x = 5. The prediction interval estimates the range in which a single new observation at x = 5 is likely to fall. The prediction interval is always wider because it includes both the estimation uncertainty (shared with the confidence interval) and the inherent scatter of individual observations around the mean.
Q: A model Y = αe^(βx)ε is proposed. Show how to linearise it and state what regression you would fit.
A: Take the natural log of both sides: log Y = log α + βx + log ε. Let a = log α and let the error term be log ε. Fit a simple linear regression of log Y on x. The intercept estimates log α (exponentiate to recover α), and the slope estimates β directly.
Q: Why might a log transformation help with heteroscedasticity?
A: When Y grows with x, larger Y values tend to have larger residuals, creating a fan-shaped residual plot. The log transformation compresses large values more than small ones, evening out the spread across the range of x and producing residuals with more constant variance.
Q: In R, how would you obtain a 95% prediction interval for y at x = 10 from a fitted quadratic model called fit?
A: predict(fit, newdata = data.frame(x = 10), interval = "prediction", level = 0.95). This returns the fitted value along with the lower and upper bounds of the prediction interval.
Model selection via AIC and BIC reappears in time-series analysis (choosing ARIMA orders), generalised linear models, and mixed-effects modelling. The confidence-vs.-prediction interval distinction carries directly into multiple regression and logistic regression (where prediction intervals take a different form). Variable transformations connect to the broader topic of generalised linear models, where a link function (such as the log) systematically relates the mean response to a linear predictor. If you later study regularisation (ridge, lasso), you will see that penalising complexity is the same core idea as parsimony, formalised differently.
AIC, BIC, Akaike information criterion, Bayesian information criterion, model selection, polynomial degree selection, t-test regression coefficients, standard error regression, confidence interval mean response, prediction interval future observation, nonlinear least squares, nls R, log transformation regression, log-log regression, elasticity, power-law model, exponential model linearisation, heteroscedasticity, variance stabilisation, poly function R, residual variance, degrees of freedom regression