Source: Principles of Statistics II, Texas A&M University
Tags: polynomial regression, nonlinear regression, least squares, R-squared, adjusted R-squared, residual analysis, overfitting, parsimony, Occam's razor, design matrix, normal equations, curve fitting, SSE, SST
Difficulty: Intermediate Prerequisites: Comfort with simple linear regression (fitting a line, interpreting slope and intercept), basic matrix notation, and the idea of residuals.
Most introductory regression work assumes a straight-line relationship between predictor and response. This topic picks up where that assumption fails. When a scatter plot curves, or when residuals show a systematic pattern rather than random scatter, a simple line is no longer adequate, and polynomial regression offers a structured way to model that curvature while still using the familiar least-squares machinery. Understanding how to fit, evaluate, and choose among polynomial models is a core skill for the rest of the course and for any applied statistics work that involves nonlinear patterns.
Polynomial regression extends simple linear regression by adding squared, cubed, and higher-power terms of the predictor, letting the fitted curve bend. The coefficients are still estimated by least squares (solved via the normal equations), and model quality is judged by R², adjusted R², AIC, and BIC, all of which balance fit against complexity. The guiding idea is parsimony: use the lowest-degree polynomial that captures the real pattern without fitting the noise.
Nonlinear relationship
A relationship between variables that cannot be adequately described by a straight line. In a scatter plot it appears as a curve, a U-shape, or some other non-straight pattern.
In simple terms, if the dots bend rather than march in a line, the relationship is nonlinear.
Polynomial regression
A regression model that includes powers of the predictor variable (x², x³, etc.) alongside the linear term. The general form is Y = β₀ + β₁x + β₂x² + ... + βₖxᵏ + ε.
Think of it as giving the regression line permission to curve by adding squared, cubed, and higher-order terms of x.
Degree (of a polynomial)
The highest power of x included in the model. A degree-2 polynomial is quadratic (one bend), degree-3 is cubic (up to two bends), and so on.
In simple terms, the degree controls how "wiggly" the fitted curve can be.
Principle of parsimony (Occam's Razor)
The modelling principle that, among models which fit the data comparably well, the simplest one should be preferred. Fewer parameters mean better interpretability and better generalisation to new data.
Think of it as: do not use a degree-5 polynomial when a degree-2 captures the pattern just as well.
Overfitting
When a model captures random noise in the sample rather than the true underlying pattern. Overfitted models look excellent on the training data but perform poorly on new observations.
In simple terms, the model has memorised the quirks of this particular dataset instead of learning the real trend.
Residual analysis
The practice of examining residuals (observed minus fitted values) to check model assumptions. Patterns or systematic curves in a residual plot suggest the model is missing something.
Think of it as the diagnostic step after fitting: if the residuals still show structure, the model needs more (or different) terms.
Design matrix (X)
The matrix used in the matrix form of regression, where each column corresponds to a power of x. For a degree-k polynomial, X has columns for 1, x, x², ..., xᵏ.
In simple terms, it is the organised table of predictor values that makes the matrix algebra work.
Normal equations
The system of linear equations X⊤Xb = X⊤y that arises from minimising the sum of squared residuals. Solving these gives the least-squares coefficient estimates.
Think of it as the set of equations you solve (or let software solve) to find the best-fitting polynomial.
Sum of Squared Residuals (SSE)
The total of all squared differences between observed and fitted values. SSE = Σ(yᵢ – ŷᵢ)². Smaller SSE means the model fits the data more closely.
In simple terms, it is the number that least squares is trying to make as small as possible.
Total Sum of Squares (SST)
The total variability in the response variable around its mean. SST = Σ(yᵢ – ȳ)². It serves as a baseline for measuring how much variability the model explains.
R² (coefficient of determination)
R² = 1 – SSE/SST. It measures the proportion of total variability in Y that is explained by the model. Ranges from 0 to 1, with higher values indicating a better fit.
In simple terms, R² tells you what fraction of the "wiggle" in Y your model accounts for.
Adjusted R² (R̄²)
A modified R² that penalises for the number of predictors. R̄² = 1 – [SSE/(n – k – 1)] / [SST/(n – 1)]. Unlike plain R², it can decrease when an added term does not improve the fit enough to justify the extra parameter.
Think of it as R² with a complexity tax: adding a useless predictor will push it down, not up.
When you plot residuals against fitted values or against the predictor and see a curve, a fan shape, or any systematic pattern, the straight-line model is missing structure in the data.
A U-shaped residual plot is a strong signal that a quadratic (or higher) term is needed.
Random scatter in the residual plot is what you want to see; it means the model has captured the systematic part of the relationship.
The general form is:
Y = β₀ + β₁x + β₂x² + ... + βₖxᵏ + ε
where ε is normally distributed with mean zero and variance σ².
Each additional power of x adds a possible bend in the curve.
Taylor's theorem provides the theoretical backing: any smooth function can be approximated arbitrarily well by a polynomial of sufficiently high degree.
Despite the nonlinear shape of the fitted curve, the model is still linear in its parameters (the βs), so ordinary least-squares estimation applies directly.
The objective is to minimise the sum of squared residuals:
f(b) = Σᵢ (yᵢ – Σⱼ bⱼxᵢʲ)²
Setting up the design matrix X with columns [1, x, x², ..., xᵏ] and the response vector y, minimisation yields the normal equations:
X⊤Xb = X⊤y
Provided X⊤X is invertible, the solution is:
b̂ = (X⊤X)⁻¹ X⊤y
This is the same matrix formula used in multiple linear regression; the only difference is that the "predictors" are powers of x rather than separate variables.
Software handles the inversion, but understanding the structure helps when diagnosing numerical issues (e.g., multicollinearity among polynomial terms).
R² always increases (or stays the same) as you add more polynomial terms, because extra parameters can never make the fit worse on the training data.
This makes R² misleading for model comparison: a degree-10 polynomial will almost always have a higher R² than a degree-2, even if degree-2 is the better model.
Adjusted R² corrects for this by penalising complexity:
R̄² = 1 – [SSE/(n – k – 1)] / [SST/(n – 1)]
If an added term does not reduce SSE enough to offset the lost degree of freedom, adjusted R² decreases.
In practice, compare adjusted R² across candidate models and favour the one where it peaks.
Polynomial regression model: Y = β₀ + β₁x + β₂x² + ... + βₖxᵏ + ε
Least-squares estimator (matrix form): b̂ = (X⊤X)⁻¹ X⊤y
R²: R² = 1 – SSE / SST
Adjusted R²: R̄² = 1 – [SSE/(n – k – 1)] / [SST/(n – 1)]
Polynomial regression is used heavily in engineering and the physical sciences. Dose-response curves in pharmacology, growth curves in biology, and the relationship between temperature and material strength are all commonly modelled with low-degree polynomials. In economics, quadratic terms capture diminishing returns (e.g., the point at which additional advertising spend stops boosting revenue).
Students often think that a higher R² always means a better model. It does not. R² can only go up as you add terms, even useless ones, so a high R² may just reflect overfitting. Use adjusted R² or information criteria instead.
Students sometimes believe polynomial regression is "nonlinear regression." The curve is nonlinear in x, but the model is linear in the parameters (the βs), and that distinction matters for estimation.
There is a tendency to forget that the design matrix columns (x, x², x³) can be highly correlated, which inflates standard errors. Centering x before computing powers often helps.
Students sometimes interpret each polynomial coefficient in isolation (e.g., "β₂ is the effect of x²"). In a polynomial, the coefficients work together to define the curve's shape; interpreting them one at a time is usually not meaningful.
⚠️ Know the formula for adjusted R² and be able to explain why it can decrease when you add a predictor.
⚠️ Be ready to identify nonlinearity from a residual plot and recommend adding a polynomial term.
⚠️ Understand why polynomial regression is still classified as a "linear model" (linear in parameters, not in x).
⚠️ The normal equations and the matrix formula for b̂ are standard exam material.
True or false: R² can decrease when you add a new polynomial term to the model.
A: False. R² can only increase or stay the same. It is adjusted R² that can decrease.
Fill in the blank: The matrix formula for the least-squares estimator is b̂ = _______.
A: (X⊤X)⁻¹ X⊤y
True or false: A degree-3 polynomial model is an example of a nonlinear regression model.
A: False. It is nonlinear in x, but linear in the parameters, so it is a linear model fitted by OLS.
Fill in the blank: The principle of parsimony says to prefer the _______ model among those that fit comparably well.
A: Simplest (fewest parameters).
Q: A residual plot from a simple linear regression shows a clear U-shape. What does this suggest, and what would you do?
A: The U-shape indicates the linear model is missing curvature in the data. Adding a quadratic term (x²) to the model would be the natural next step. After fitting the quadratic model, check the new residual plot to see whether the pattern has been resolved.
Q: Explain why adjusted R² is preferred over R² when comparing polynomial models of different degrees.
A: R² never decreases when a term is added, so it will always favour the more complex model regardless of whether the extra term captures real structure or just noise. Adjusted R² penalises each additional parameter by adjusting for degrees of freedom, so it can decrease if an added term is not worth its complexity cost. This makes it a fairer comparison tool.
Q: Write the polynomial regression model for a cubic (degree-3) fit and identify the design matrix columns.
A: The model is Y = β₀ + β₁x + β₂x² + β₃x³ + ε. The design matrix X has four columns: a column of ones (for the intercept), x, x², and x³, with one row per observation.
Q: What does Taylor's theorem have to do with polynomial regression?
A: Taylor's theorem guarantees that any sufficiently smooth function can be approximated to arbitrary accuracy by a polynomial of high enough degree. This provides the theoretical justification for using polynomial models to approximate unknown nonlinear relationships.
Q: A student fits a degree-8 polynomial and reports R² = 0.997. Should you trust this model? Why or why not?
A: A very high R² with a high-degree polynomial is a classic sign of overfitting. The model may be fitting noise rather than the true pattern. Check adjusted R², AIC, or BIC, and consider whether a lower-degree polynomial achieves a comparable fit. Also examine the residual plot and think about whether a degree-8 relationship makes substantive sense.
This material connects directly to simple linear regression (the special case where k = 1) and to multiple regression more broadly, since polynomial regression is just multiple regression with constructed predictors (x², x³, etc.). The model-selection ideas here (parsimony, adjusted R²) reappear in variable selection for multiple regression and in regularisation methods like ridge and lasso. Understanding residual analysis at this stage also prepares you for diagnostics in logistic regression and generalised linear models later in the course.
polynomial regression, quadratic regression, cubic regression, nonlinear curve fitting, least squares estimation, normal equations, design matrix, R-squared, adjusted R-squared, coefficient of determination, overfitting, parsimony, Occam's razor, residual analysis, residual plot, Taylor's theorem, model complexity, degrees of freedom, sum of squared residuals, SSE, SST, polynomial degree, regression diagnostics, curve fitting statistics