Difficulty: Intermediate | Prerequisites: Simple linear regression, least squares estimation, residual analysis basics.
Polynomial regression sits at the boundary between simple linear modelling and fully nonlinear approaches. When a scatterplot or residual plot tells you the straight-line fit is not enough, polynomial terms let you capture curvature without abandoning the least-squares machinery you already know. This material builds directly on simple linear regression (OLS, residual diagnostics, R-squared) and extends it to handle curves, bends, and more complex shapes in the data. If you are comfortable fitting a line and interpreting residuals, you are ready for this.
Polynomial regression adds powers of x (x², x³, etc.) to a linear model so it can follow curved relationships. Despite the nonlinear shape, the model is still linear in its parameters, so ordinary least squares still works. The main danger is overfitting: higher-degree polynomials fit training data better but may perform terribly on new data, so you need model-selection criteria (adjusted R², AIC, BIC) to choose the right degree.
Nonlinearity (in regression)
A relationship between predictor and response that a straight line cannot adequately describe. Think of it as the data bending or curving in a way that leaves systematic patterns in your residuals.
Polynomial regression
A regression model that includes powers of the predictor variable (x², x³, ..., x^k) alongside the linear term. In simple terms, you are fitting a curve instead of a straight line, but you still use least squares to estimate the coefficients.
Residual analysis
The practice of examining residuals (observed minus fitted values) for patterns. If the residuals show curvature or systematic fanning rather than random scatter, the current model is missing something.
Design matrix (X)
The matrix whose columns correspond to the predictor terms (1, x, x², ..., x^k) and whose rows correspond to observations. This is the structure you feed into the normal equations for least squares.
Normal equations
The system of equations X'Xb = X'y that, when solved, yield the least squares coefficient estimates. In simple terms, these are the conditions that make the sum of squared residuals as small as possible.
Overfitting
When a model captures noise in the training data rather than the true underlying pattern, producing excellent in-sample fit but poor predictions on new data. A polynomial of degree 20 will hit every data point in a small sample, but its predictions between and beyond those points will be wild.
R-squared (R²)
The proportion of variance in the response explained by the model. It always goes up (or stays the same) as you add more terms, which is precisely why it cannot be trusted on its own for model selection.
Adjusted R-squared (R²_adj)
A modified R² that penalises additional parameters. Unlike plain R², it can decrease when an added term does not improve the model enough to justify its complexity.
AIC (Akaike Information Criterion)
A model-selection metric that balances goodness of fit against the number of parameters. Lower AIC indicates a better trade-off. Think of it as a score that rewards accuracy but charges you for every extra parameter.
BIC (Bayesian Information Criterion)
Similar to AIC but with a heavier penalty for complexity, especially in larger samples. It tends to favour simpler models than AIC does.
Degrees of freedom (n - k - 1)
In a polynomial regression of degree k with n observations, this is the number of independent pieces of information left after estimating k + 1 coefficients. Used in t-tests and confidence intervals for the coefficients.
The primary diagnostic is a residual plot: plot residuals against fitted values or against the predictor.
Random scatter around zero means the linear model is adequate.
Curvature, systematic waves, or fan-shaped spread signals that a more flexible model is needed.
Polynomial regression is the most common first step when curvature appears, because it extends the existing linear framework rather than replacing it.
General form: Y = β₀ + β₁x + β₂x² + ... + β_k x^k + ε
Despite the curved shape, this is still "linear in the parameters" because each β coefficient enters the equation as a simple multiplier. The nonlinearity is in x, not in the β's.
Taylor's theorem provides the theoretical backing: any smooth function can be approximated to arbitrary accuracy by a polynomial of sufficiently high degree.
The objective function is the sum of squared residuals: f(b₀, b₁, ..., b_k) = Σ(yᵢ - (b₀ + b₁xᵢ + ... + b_k xᵢ^k))²
Setting partial derivatives to zero produces the normal equations, which in matrix form are: X'Xb = X'y
The solution (assuming X'X is invertible): b̂ = (X'X)⁻¹ X'y
This is the same mechanical process as simple linear regression; the only difference is that X now has columns for x², x³, etc.
R² always increases (or stays flat) as you add polynomial terms, so it will never tell you to stop adding complexity.
R²_adj penalises additional parameters and can decrease when a new term does not earn its keep.
AIC balances fit against complexity; lower is better.
BIC applies a stronger complexity penalty than AIC, especially for large n, and generally prefers simpler models.
Practical pattern: as polynomial degree rises, R² climbs towards 1, but R²_adj and BIC reach a peak (or trough, for BIC) at the best degree and then worsen.
Polynomial model: Y = β₀ + β₁x + β₂x² + ... + β_k x^k + ε
Least squares objective: f(b₀, ..., b_k) = Σ(yᵢ - (b₀ + b₁xᵢ + ... + b_k xᵢ^k))²
Normal equations (matrix form): X'Xb = X'y
Coefficient estimates: b̂ = (X'X)⁻¹ X'y
Standard error of β̂ᵢ: SE(β̂ᵢ) = σ̂ × √(cᵢ₊₁)
where cᵢ₊₁ is the (i+1)-th diagonal element of (X'X)⁻¹ and σ̂ is the estimated residual standard deviation.
t-statistic for H₀: βⱼ = 0: t = β̂ⱼ / SE(β̂ⱼ), with n - k - 1 degrees of freedom.
Confidence interval for βⱼ: β̂ⱼ ± t(n-k-1, α/2) × SE(β̂ⱼ)
Confidence interval for the mean response at x₀: μ̂(x₀) ± t(n-k-1, α/2) × σ̂ × √(x₀'(X'X)⁻¹x₀)
Prediction interval for a new observation at x₀: μ̂(x₀) ± t(n-k-1, α/2) × σ̂ × √(1 + x₀'(X'X)⁻¹x₀)
Each coefficient gets its own t-test. The null hypothesis H₀: βⱼ = 0 asks whether the j-th polynomial term contributes to the model.
The t-statistic uses the coefficient estimate divided by its standard error, and follows a t-distribution with n - k - 1 degrees of freedom.
Confidence intervals for coefficients take the familiar form: estimate ± critical value × standard error.
A confidence interval for the mean response at x₀ captures uncertainty about where the regression surface sits on average.
A prediction interval for a single new observation at x₀ is always wider because it adds the inherent variability (σ²) of a new data point on top of estimation uncertainty.
Both intervals widen as x₀ moves further from the centre of the observed data.
Fit the model: fit <- lm(y ~ poly(x, degree, raw=TRUE))
View coefficient estimates and significance tests: summary(fit)
Get confidence intervals for coefficients: confint(fit)
Predict the mean response with a confidence interval: predict(fit, newdata=data.frame(x=x0), interval="confidence")
Predict a new observation with a prediction interval: predict(fit, newdata=data.frame(x=x0), interval="prediction")
The raw=TRUE argument in poly() uses raw polynomial terms (x, x², x³, ...) rather than orthogonal polynomials, which keeps the coefficients directly interpretable.
Polynomial regression appears in engineering (modelling stress-strain curves that bend), economics (fitting price or demand trends that level off or accelerate), biology (dose-response curves), and environmental science (seasonal or cyclical patterns). Anywhere a relationship bends, polynomial terms are a natural first tool.
Students often think "nonlinear model" means "nonlinear in the parameters." Polynomial regression is nonlinear in x but linear in the β's, which is why OLS still applies. The distinction matters for estimation.
Students sometimes believe higher R² always means a better model. It does not. R² cannot decrease when you add a term, so it rewards complexity without limit. Use adjusted R², AIC, or BIC instead.
Students frequently confuse confidence intervals and prediction intervals. The confidence interval is for the average response; the prediction interval is for a single new point. The prediction interval is always wider.
Students occasionally forget the hierarchy principle: if you include x³ in a model, you should generally also include x² and x, even if their individual p-values are not significant.
⚠️ Know when to choose polynomial regression vs. a simple linear model. If residual plots show curvature, the linear model is inadequate.
⚠️ Be able to write down the design matrix X for a given polynomial degree and explain what each column represents.
⚠️ Understand why R² is unreliable for choosing polynomial degree and which alternatives (R²_adj, AIC, BIC) correct for this.
⚠️ Be ready to compute or interpret a t-test for a polynomial coefficient and explain what "βⱼ = 0" means in context.
⚠️ Know the difference between a confidence interval for the mean response and a prediction interval for a new observation, and be able to state which is wider and why.
True or false: Polynomial regression requires nonlinear least squares to estimate its coefficients. False. It is linear in the parameters, so ordinary least squares works.
Fill in the blank: As polynomial degree increases, R² always ________, but R²_adj may ________. Increases (or stays the same); decrease.
True or false: A prediction interval at x₀ is narrower than a confidence interval at x₀. False. The prediction interval is always wider because it accounts for the variability of a new observation.
Fill in the blank: The degrees of freedom for the t-test on a polynomial coefficient in a degree-k model with n observations is ________. n - k - 1.
Q: Why does plotting residuals against fitted values help detect nonlinearity?
A: If the true relationship is nonlinear but you have fit a linear model, the residuals will show a systematic curved pattern rather than random scatter. This pattern reveals the structure the linear model missed.
Q: Write the general form of a degree-3 polynomial regression model.
A: Y = β₀ + β₁x + β₂x² + β₃x³ + ε
Q: Explain why R² is a poor criterion for selecting polynomial degree.
A: R² can only increase (or stay the same) as you add polynomial terms, so it will always favour the most complex model. It does not account for the cost of additional parameters and will lead you towards overfitting.
Q: In R, what does poly(x, 3, raw=TRUE) produce?
A: A matrix with three columns containing x, x², and x³ (the raw polynomial terms), which are used as predictors in the regression.
Q: A prediction interval is wider than a confidence interval at the same x₀. Why?
A: The confidence interval captures uncertainty about the mean response (estimation error only). The prediction interval must also include the random variation of a new individual observation (σ²), making it wider.
This material connects directly to simple linear regression, since polynomial regression is an extension that uses the same estimation framework with additional predictor columns. It also links to multiple regression more broadly, as each polynomial term (x², x³) acts like an additional predictor. The model-selection criteria (AIC, BIC) reappear in variable selection, mixed models, and time-series analysis throughout the course.
polynomial regression, nonlinear relationships, curvilinear regression, quadratic regression, cubic regression, least squares, OLS, design matrix, normal equations, R-squared, adjusted R-squared, AIC, Akaike information criterion, BIC, Bayesian information criterion, overfitting, model selection, residual analysis, residual plots, t-test for coefficients, confidence interval, prediction interval, Taylor's theorem, poly() in R, polynomial degree selection, Principles of Statistics II, Texas A&M