Source: STAT 212 Lecture Notes, Texas A&M University Difficulty: Intermediate | Prerequisites: Simple linear regression (STAT 212 Ch. 1 Part A), basic calculus, matrix notation
Tags: polynomial regression, nonlinearity, residual plots, least squares, normal equations, design matrix, Occam's razor, parsimony, Taylor's theorem, STAT 212
This material picks up where simple linear regression left off. You already know how to fit a straight line, Y = β₀ + β₁x + ε. The problem is that real data often curve, and a straight line cannot capture that curvature. Polynomial regression is the natural next step: replace the straight line with a polynomial of degree k, giving you a flexible family of models that still use least squares. Everything you learned about residuals, SSE, R², and inference in the simple case carries over here with minor adjustments. If you are comfortable with simple linear regression, you are ready for this.
When a straight line does not fit well, polynomial regression lets you model Y as a function of x, x², x³, and so on. The coefficients are still estimated by least squares, and the normal equations can be written in matrix form as XᵀXb = Xᵀy. The principle of parsimony says you should use the lowest-degree polynomial that fits the data adequately.
Nonlinearity
A relationship between x and Y that is not adequately described by a straight line. In simple terms, your scatterplot curves rather than following a flat trend.
Residual plot
A plot of residuals (eᵢ = yᵢ − ŷᵢ) against x. If the residuals show a systematic pattern (such as a wave or curve), the linear model is missing something. Think of it as a diagnostic check: random scatter means the model is fine, a pattern means it is not.
Smoothed residuals
A smooth curve fitted through the residual plot to make systematic patterns easier to spot. If the smoother deviates noticeably from zero, nonlinearity is likely present.
Polynomial regression model
Y = β₀ + β₁x + β₂x² + ⋯ + βₖxᵏ + ε, where ε ~ N(0, σ²). A generalisation of simple linear regression that allows curvature by including higher powers of x.
Degree (of a polynomial)
The highest power of x in the model. A degree-1 polynomial is a straight line, degree-2 is quadratic, degree-3 is cubic, and so on.
Principle of parsimony (Occam's razor)
"Use the simplest model, the one with the fewest parameters, that still fits the data adequately." In simple terms, do not make the model more complicated than it needs to be.
Taylor's theorem
A mathematical result stating that any smooth function can be approximated arbitrarily well by a polynomial of sufficiently high degree. This is the theoretical justification for using polynomials as a flexible model class.
Normal equations
The system of linear equations obtained by setting the partial derivatives of the least squares objective to zero. They have the matrix form XᵀXb = Xᵀy.
Design matrix (X)
The n × (k + 1) matrix whose rows contain 1, xᵢ, xᵢ², …, xᵢᵏ for each observation. It organises all the predictor information needed for the matrix formulation.
Least squares estimates (β̂)
The values b₀, b₁, …, bₖ that minimise the sum of squared residuals. In matrix form: β̂ = (XᵀX)⁻¹Xᵀy.
Fit a straight line to the data and compute residuals eᵢ = yᵢ − ŷᵢ.
Plot residuals against x. If the plot shows a random scatter around zero, the straight line is adequate.
If you see a systematic curve or wave, the relationship is nonlinear and a more flexible model is needed.
Overlaying a smoother (such as loess) on the residual plot makes patterns easier to detect.
The lecture example uses data generated from Yⱼ = 20 + 10xⱼ + 0.25 sin(4πxⱼ) + εⱼ. The residuals from a straight line fit show a clear sinusoidal pattern, confirming the line misses the nonlinear component.
The simple model Y = β₀ + β₁x + ε is a special case of Y = f(x) + ε where f is an arbitrary function.
In practice, allowing f to be completely arbitrary gives too many parameters to estimate. We need to restrict the class of models.
Polynomials are a good choice because they are flexible (Taylor's theorem guarantees they can approximate smooth functions) and because least squares estimation leads to linear equations, which are straightforward to solve.
The objective function to minimise is: f(b₀, b₁, …, bₖ) = Σᵢ₌₁ⁿ (yᵢ − Σⱼ₌₀ᵏ bⱼxᵢʲ)²
Setting each partial derivative ∂f/∂bⱼ = 0 for j = 0, 1, …, k produces k + 1 equations.
These are the normal equations, and they are linear in b₀, b₁, …, bₖ, which is why polynomial regression is computationally convenient.
Define b = [b₀, b₁, …, bₖ]ᵀ, the vector of coefficients.
Define X as the n × (k + 1) design matrix with row i equal to [1, xᵢ, xᵢ², …, xᵢᵏ].
Define y = [y₁, y₂, …, yₙ]ᵀ.
The normal equations become XᵀXb = Xᵀy.
If XᵀX is invertible, the solution is b = (XᵀX)⁻¹Xᵀy = β̂.
Polynomial regression model: Y = β₀ + β₁x + β₂x² + ⋯ + βₖxᵏ + ε, where ε ~ N(0, σ²)
Least squares objective: f(b₀, …, bₖ) = Σᵢ₌₁ⁿ (yᵢ − Σⱼ₌₀ᵏ bⱼxᵢʲ)²
Normal equations (matrix form): XᵀXb = Xᵀy
Least squares solution: β̂ = (XᵀX)⁻¹Xᵀy
Residuals: eᵢ = yᵢ − ŷᵢ, where ŷᵢ = β̂₀ + β̂₁xᵢ + ⋯ + β̂ₖxᵢᵏ
SSE, SSR, SST, R²: SSE = Σeᵢ², SST = Σ(yᵢ − ȳ)², SSR = Σ(ŷᵢ − ȳ)² SST = SSR + SSE R² = SSR / SST = 1 − SSE / SST
Variance estimate: σ̂² = MSE = SSE / (n − k − 1)
Polynomial regression is used whenever a response variable has a curved relationship with a predictor that cannot be captured by a straight line. In engineering, for example, the deflection of a beam under load is often modelled as a polynomial function of position along the beam. In economics, cost functions and production functions frequently exhibit curvature that a quadratic or cubic polynomial can approximate.
Students often think polynomial regression is a fundamentally different method from simple linear regression. It is not. It is a direct extension: you still minimise the sum of squared residuals, and the normal equations are still linear in the unknown coefficients.
Some students believe that because the model includes x², x³, etc., the estimation procedure becomes "nonlinear." The model is nonlinear in x, but it is linear in the parameters β₀, β₁, …, βₖ. That linearity in parameters is what makes the normal equations linear.
A common mistake is to look at a scatterplot of y vs. x and conclude a straight line is fine because R² is high. A high R² does not rule out nonlinearity. Always check the residual plot.
Students sometimes forget that the denominator for MSE in polynomial regression is n − k − 1 (not n − 2 as in simple regression). The "k + 1" accounts for the k + 1 parameters being estimated.
⚠️ You must know how to write the design matrix X for a given polynomial degree k. This is a very common exam question.
⚠️ The formula σ̂² = SSE / (n − k − 1) is frequently tested. Remember that k is the polynomial degree, so a cubic model has k = 3 and the denominator is n − 4.
⚠️ Be able to explain why polynomial regression uses least squares and why the resulting equations are linear (even though the model is nonlinear in x).
⚠️ Residual plots as a diagnostic for nonlinearity come up regularly. Know what a "good" residual plot (random scatter) looks like versus a "bad" one (systematic pattern).
True or false: A polynomial regression model with degree k = 2 has three parameters to estimate.
A: True (β₀, β₁, β₂).
Fill in the blank: The matrix form of the normal equations is XᵀXb = ______.
A: Xᵀy
True or false: R² for a polynomial model is always at least as large as R² for a straight-line model fitted to the same data.
A: True. R² cannot decrease when you add more polynomial terms.
Fill in the blank: The denominator of MSE for a degree-k polynomial fitted to n data points is ______.
A: n − k − 1
Q: What is the principle of parsimony, and why does it matter in polynomial regression?
A: The principle of parsimony (Occam's razor) says to use the simplest model that still fits the data adequately. In polynomial regression, this means choosing the lowest degree k that captures the essential shape of the relationship, rather than fitting a high-degree polynomial that may overfit noise.
Q: Suppose you fit a straight line to data and notice a clear U-shaped pattern in the residual plot. What does this suggest, and what would you try next?
A: The U-shaped pattern indicates the true relationship between x and Y is nonlinear, so the straight line is inadequate. The natural next step is to fit a quadratic (degree 2) polynomial and check whether the residual plot improves.
Q: Write the design matrix X for a quadratic model fitted to three observations with x-values 1, 2, and 3.
A: X = [[1, 1, 1], [1, 2, 4], [1, 3, 9]]. Each row is [1, xᵢ, xᵢ²].
Q: Why is the normal equation system for polynomial regression called "linear" when the model contains x², x³, and so on?
A: The equations are linear in the unknown coefficients b₀, b₁, …, bₖ. The powers of x are known values computed from the data, not unknowns. It is linearity in the parameters that matters for the normal equations.
Q: For a degree-4 polynomial fitted to 50 data points, what is the denominator of MSE?
A: n − k − 1 = 50 − 4 − 1 = 45.
This material connects directly to multiple regression (coming later in STAT 212), because polynomial regression is a special case where the "multiple predictors" are just powers of a single variable x. The matrix formulation XᵀXb = Xᵀy is identical.
It also connects to model diagnostics: the residual-plot techniques introduced here for detecting nonlinearity will be used throughout the rest of the course for checking other model assumptions (constant variance, normality).
polynomial regression, curvilinear regression, nonlinear relationship, residual analysis, residual plot, smoothed residuals, loess smoother, normal equations, design matrix, least squares estimation, matrix form regression, Occam's razor, parsimony, Taylor's theorem, polynomial degree, SSE, SSR, SST, R-squared, MSE, STAT 212, Texas A&M, principles of statistics II