Source: Lecture Notes, Chapter 1 (Part C) – Dr. Abhishek Chakrabortty, Texas A&M University
Tags: multiple linear regression, MLR, multiple regression model, least squares estimation, normal equations, design matrix, predictors, covariates, response variable, interaction effect, quadratic terms, polynomial regression, log-log regression, STAT 212
Difficulty: Intermediate Prerequisites: Simple linear regression (Chapter 1 Parts A and B), basic matrix algebra, logarithms.
Multiple linear regression extends simple linear regression to settings where more than one predictor might influence the response. Where simple regression draws a line through a cloud of points, multiple regression fits a surface (or hyperplane) through higher-dimensional data. This is the workhorse tool for most applied statistical modelling, and nearly everything from Chapter 1 Parts A and B carries over, just with matrices replacing scalar algebra. You need to be comfortable with the simple-regression assumptions and the idea of least squares before tackling this material.
Multiple linear regression models a response Y as a linear function of two or more predictors plus a normally distributed error term. The least squares estimates are obtained by solving the normal equations in matrix form. The model is more flexible than it first appears, because the "variables" in the equation can themselves be transformations (interactions, polynomials, or logs) of the original measured quantities.
Multiple regression
A regression model with one dependent variable (response) Y and more than one independent variable (predictor/covariate). In simple terms, you are trying to explain Y using several x variables at once rather than just one.
Dependent variable / response / outcome (Y)
The variable you are trying to predict or explain. Think of it as the thing you measure and want to understand.
Independent variables / predictors / covariates (x₁, x₂, ..., xₖ)
The variables you believe influence Y. In simple terms, these are the inputs to your model.
Error term (ε)
The random component capturing everything the model does not explain. Assumed to satisfy E(ε) = 0, Var(ε) = σ², and ε ~ N(0, σ²), with errors independent of one another and of the predictors.
Interaction effect
A term formed by multiplying two predictors together (e.g. x₃ = x₁·x₂). It captures the idea that the effect of one predictor on Y may depend on the level of another predictor.
Quadratic effect
A term formed by squaring a predictor (e.g. x₃ = x₁²). It allows the regression surface to curve rather than remaining a flat plane.
Design matrix (X)
The n × (k + 1) matrix whose first column is all ones (for the intercept) and whose remaining columns hold the observed values of x₁ through xₖ. This is the matrix that organises all predictor data for the least squares calculation.
Normal equations
The system of equations (XᵀX)b = Xᵀy that, when solved, yields the least squares coefficient estimates. Think of it as the matrix version of "set the partial derivatives to zero."
Least squares estimates (β̂)
The vector β̂ = (XᵀX)⁻¹Xᵀy. These are the coefficient values that minimise the sum of squared residuals.
Log-log linear regression
A technique where both the response and the predictors are log-transformed before fitting a standard linear model. Coefficients in a log-log model are interpreted as elasticities: a 1% change in a predictor corresponds to approximately a β% change in the response.
Elasticity (in log-log models)
The percentage change in the response variable associated with a 1% change in a predictor. In simple terms, if β̂₁ = 0.96, a 1% increase in that predictor leads to roughly a 0.96% increase in the response.
The general form is:
Y = β₀ + β₁x₁ + β₂x₂ + ... + βₖxₖ + ε
The expected (average) value of Y given the predictors is:
E(Y) = β₀ + β₁x₁ + β₂x₂ + ... + βₖxₖ
All expectations and variances here are conditional on the predictor values x₁, ..., xₖ.
The standard assumptions on ε are:
E(ε) = 0
Var(ε) = σ² (constant, does not depend on x₁, ..., xₖ)
ε ~ N(0, σ²)
εᵢ ⊥ εⱼ for all i ≠ j (errors are independent of one another)
Two situations arise:
Situation I: k equals the number of separate measured independent variables. Each experimental unit provides k + 1 measurements (k predictors plus y).
Situation II: The number of distinct measured variables is p, which is less than k. The extra terms come from transformations, interactions, or polynomial terms.
For example, with only p = 2 measured variables (x₁ and x₂), you could fit:
Model M1: Y = β₀ + β₁x₁ + β₂x₂ + β₃x₁x₂ + ε (here k = 3, with x₃ = x₁x₂ as an interaction)
Model M2: Y = β₀ + β₁x₁ + β₂x₂ + β₃x₁² + β₄x₂² + ε (here k = 4, with quadratic terms)
The function f(x₁, x₂) = β₀ + β₁x₁ + β₂x₂ describes a flat plane with no curvature. If you suspect the surface of averages E(Y | x₁, x₂) curves, models M1 or M2 (with interaction or quadratic terms) are more appropriate.
Data arrive as (xᵢ₁, xᵢ₂, ..., xᵢₖ, yᵢ) for i = 1, ..., n.
Define the objective function:
f(b₀, b₁, ..., bₖ) = Σᵢ₌₁ⁿ [yᵢ - (b₀ + b₁xᵢ₁ + ... + bₖxᵢₖ)]²
We choose b₀, b₁, ..., bₖ to minimise f. Setting the partial derivatives to zero gives:
∂f/∂bⱼ = 0 ⟺ Σᵢ₌₁ⁿ [yᵢ - (b₀ + b₁xᵢ₁ + ... + bₖxᵢₖ)]xᵢⱼ = 0
for j = 0, 1, ..., k, with xᵢ₀ = 1 for every i.
Define:
X = n × (k+1) design matrix, with (j+1)st column [x₁ⱼ, x₂ⱼ, ..., xₙⱼ]ᵀ (the first column is all ones)
b = [b₀, b₁, ..., bₖ]ᵀ, a (k+1) × 1 vector
y = [y₁, y₂, ..., yₙ]ᵀ, an n × 1 vector
The normal equations are:
(XᵀX)b = Xᵀy
The solution is:
β̂ = (XᵀX)⁻¹Xᵀy
This requires n > k (more observations than parameters) for (XᵀX) to be invertible.
In R, with response y and predictors x1, x2, x3:
summary(lm(y ~ x1 + x2 + x3))
Sometimes a multiplicative model is more natural than a linear one. For the mesquite tree data, leaf weight (z) should be roughly proportional to canopy volume, which is proportional to the product of canopy dimensions.
The multiplicative model: Z = β₀* · u₁^β₁ · u₂^β₂ · u₃^β₃ · u₄^β₄ · ε'
Taking logarithms of both sides converts this to a standard linear model:
y = β₀ + β₁x₁ + β₂x₂ + β₃x₃ + β₄x₄ + ε
where y = log(z), xⱼ = log(uⱼ), β₀ = log(β₀*), and ε = log(ε').
The variables for the mesquite data are:
z = leafwt: total weight (grams) of photosynthetic material
u₁ = diam1: canopy diameter along the longest axis (metres)
u₂ = diam2: canopy diameter along the shortest axis (metres)
u₃ = totht: total height of tree (metres)
u₄ = canht: canopy height (metres)
The fitted coefficients from R: β̂₀ = 4.3043, β̂₁ = 0.9579, β̂₂ = 1.0194, β̂₃ = 1.1650, β̂₄ = −0.6040.
Because this is a log-log model, each coefficient is an elasticity. For instance, a 1% increase in diam1 leads to approximately a 0.96% increase in leaf weight.
The negative coefficient on x₄ (canopy height) is counterintuitive and worth noting, as it implies that, holding the other variables fixed, greater canopy height is associated with lower leaf weight.
If a 95% prediction interval for y = log(z) is (y₁, y₂), then a 95% prediction interval for z (the original scale) is (e^y₁, e^y₂). The log transformation preserves the coverage probability.
Item | Formula |
|---|---|
MLR model | Y = β₀ + β₁x₁ + ... + βₖxₖ + ε |
Error assumptions | ε ~ N(0, σ²), independent across observations |
Objective function | f(b) = Σ(yᵢ - b₀ - b₁xᵢ₁ - ... - bₖxᵢₖ)² |
Normal equations | (XᵀX)b = Xᵀy |
LS estimates | β̂ = (XᵀX)⁻¹Xᵀy |
Log-log transformation | y = log(z), xⱼ = log(uⱼ) |
The mesquite tree example is a good illustration: ecologists use multiple regression to predict leaf biomass from easy-to-measure canopy dimensions, avoiding the need to physically harvest trees. This same logic applies in agriculture (estimating crop yield from satellite measurements), medicine (predicting blood pressure from age, weight, and cholesterol), and engineering (modelling product quality from multiple process variables).
Students often assume that "linear" in "multiple linear regression" means the relationship between Y and each predictor must be a straight line. It does not. The model is linear in the parameters (the βs), but the predictors themselves can be squared, logged, or multiplied together.
Students sometimes forget that the design matrix X includes a column of ones for the intercept. Leaving it out changes the model and the estimates.
A common confusion: believing you need to measure k separate variables to have k terms in the model. You can generate interaction and polynomial terms from a smaller set of measured variables.
Students occasionally think that a negative coefficient always means the predictor "hurts" the response. In a model with correlated predictors, a negative coefficient for one variable can simply reflect that its information is already captured by the others.
⚠️ Be able to write out the design matrix X for a given dataset and model specification, including the intercept column.
⚠️ Know how to go from the normal equations to β̂ = (XᵀX)⁻¹Xᵀy.
⚠️ Understand the distinction between the number of measured variables (p) and the number of terms (k) in a model with interactions or polynomial terms.
⚠️ Know how to interpret coefficients in a log-log model as elasticities.
⚠️ Know that prediction intervals on the log scale can be back-transformed by exponentiating the endpoints.
True or False: In a multiple linear regression model, the predictors must be independent of each other.
Fill in the blank: The least squares estimates are found by solving the _______ equations in matrix form.
True or False: If you have p = 2 measured variables but include an interaction term, then k = 3 in the model.
Fill in the blank: In a log-log regression model, the coefficients are interpreted as _______.
True or False: The design matrix X has dimensions n × k.
Answers: 1. False (predictors can be correlated; this affects interpretation but not estimation). 2. Normal. 3. True. 4. Elasticities. 5. False (it is n × (k + 1), because of the intercept column).
Q: Write the multiple regression model for predicting systolic blood pressure (y) from age (x₁), weight (x₂), and cholesterol level (x₃). State the assumptions on the error term.
A: Y = β₀ + β₁x₁ + β₂x₂ + β₃x₃ + ε, where E(ε) = 0, Var(ε) = σ² (constant, not depending on x₁, x₂, x₃), ε ~ N(0, σ²), and errors are independent across observations.
Q: Suppose you have two measured predictors x₁ and x₂. Write a model that includes both linear terms, their interaction, and quadratic terms for each. How many parameters (excluding the intercept) does this model have?
A: Y = β₀ + β₁x₁ + β₂x₂ + β₃x₁x₂ + β₄x₁² + β₅x₂² + ε. There are 5 parameters (β₁ through β₅) excluding the intercept, so k = 5.
Q: In a log-log regression on the mesquite data, β̂₁ = 0.9579. Interpret this value.
A: A 1% increase in diam1 (canopy diameter along the longest axis) is associated with approximately a 0.96% increase in leaf weight, holding the other predictors constant.
Q: If a 95% prediction interval for log(leaf weight) is (5.58, 7.53), what is the corresponding 95% prediction interval for leaf weight in grams?
A: Exponentiate both endpoints: (e^5.58, e^7.53) = (265.1, 1863.1) grams.
Q: What R command would you use to fit a multiple regression of y on predictors x1, x2, x3, and x4, and view the summary output?
A: summary(lm(y ~ x1 + x2 + x3 + x4))
This material extends directly from simple linear regression and polynomial regression covered earlier in Chapter 1. The matrix formulation here (β̂ = (XᵀX)⁻¹Xᵀy) is the general version of the formulas you used in the two-predictor case. The ideas of SSE, R², and residual analysis all carry over and are developed further in Part 2 of these notes (Inference and the F-Test) and Part 3 (Model Selection and Diagnostics).
Related Terms / Search Tags: multiple linear regression, MLR, multiple regression, least squares, OLS, normal equations, design matrix, interaction term, quadratic regression, polynomial regression, log-log regression, multiplicative model, elasticity, STAT 212, covariates, predictors, response variable, conditional expectation, error variance, R command lm, mesquite tree data