Source: Comprehensive Overview of Statistical Analysis Techniques (Texas A&M University)
Tags: multiple linear regression, MLR, regression analysis, OLS, ordinary least squares, dependent variable, independent variable, predictors, R-squared, AIC, BIC, model selection, F-test, t-test, confidence interval, prediction interval, residuals, STAT 101
Difficulty: Intermediate | Prerequisites: Basic simple linear regression, familiarity with hypothesis testing and p-values.
Multiple linear regression extends simple linear regression to handle more than one predictor variable at a time. Where simple regression draws a line through a scatterplot, MLR fits a surface (or hyperplane) through higher-dimensional data. This is the workhorse technique for most applied research: whenever you need to predict an outcome from several measured quantities, or understand which factors matter most while controlling for the others, MLR is typically the first tool you reach for. You should already be comfortable with the idea of a regression equation, residuals, and what R-squared means in the single-predictor case.
Multiple linear regression models one outcome variable as a linear combination of two or more predictor variables, each with its own coefficient. You estimate those coefficients (usually via ordinary least squares), check that your assumptions hold, then use the model to make predictions or test which predictors matter.
Multiple Linear Regression (MLR)
A statistical technique that models the relationship between one dependent (response) variable and two or more independent (predictor) variables using a linear equation. In simple terms, it answers "how do several factors, taken together, predict an outcome?"
Dependent variable (Y)
The outcome you are trying to predict or explain. Think of it as the thing that changes in response to everything else.
Independent variables (X₁, X₂, ..., Xₖ)
The predictor variables, also called explanatory variables or features. These are the inputs you believe influence the outcome.
Intercept (β₀)
The predicted value of Y when every predictor equals zero. Think of it as the baseline starting point of your model before any predictor has an effect.
Regression coefficients (β₁, β₂, ..., βₖ)
Each coefficient tells you how much Y is expected to change for a one-unit increase in that predictor, holding all other predictors constant. In simple terms, they are the "weights" assigned to each input.
Error term (ε)
The difference between what the model predicts and what is observed. It captures everything the model does not explain: measurement noise, omitted variables, random variation.
Ordinary Least Squares (OLS)
The standard method for estimating regression coefficients. OLS finds the set of coefficients that minimises the sum of squared residuals (the squared gaps between predicted and observed values).
R-squared (R²)
The proportion of variability in Y that the model explains. Ranges from 0 to 1. Think of it as a percentage score for how well your predictors account for the outcome's variation.
Akaike Information Criterion (AIC)
A model-comparison metric that balances goodness of fit against model complexity. Lower AIC is better. It penalises you for adding predictors that do not earn their keep.
Bayesian Information Criterion (BIC)
Similar to AIC but with a heavier penalty for extra parameters, especially in larger samples. Also lower-is-better.
F-test (for model utility)
A hypothesis test that checks whether the model as a whole explains significantly more variability than a model with no predictors at all. In simple terms, it asks "does this collection of predictors, taken together, tell us anything useful?"
t-test (for individual coefficients)
A hypothesis test applied to each coefficient individually, checking whether that predictor makes a statistically significant contribution after accounting for the others.
Confidence interval (for predictions)
An interval estimate for the mean response at a given set of predictor values. It captures uncertainty about the average outcome.
Prediction interval
An interval estimate for a single new observation. Wider than the confidence interval because it also accounts for individual-level variability around the mean.
The general form is:
Y = β₀ + β₁X₁ + β₂X₂ + ... + βₖXₖ + ε
Y is the response.
Each Xᵢ is a predictor.
Each βᵢ is the change in Y per one-unit change in Xᵢ, with all other predictors held fixed.
ε is the residual error for each observation.
Data collection: Gather observations and identify which variables might serve as predictors. Quality of input data constrains everything downstream.
Model specification: Decide which predictors to include. This can be guided by theory, prior research, or exploratory analysis.
Parameter estimation: Use OLS (or another estimation method) to compute β₀ through βₖ. OLS minimises the total squared distance between observed and predicted values.
Model diagnostics: After fitting, check the four key assumptions (see below). Violations can invalidate your inferences.
Four conditions need to hold for OLS estimates to be reliable:
Linearity: The relationship between each predictor and the response is linear (after any transformations).
Independence: Observations are independent of one another. Violated, for example, when data is collected over time and sequential values are correlated.
Homoscedasticity: The spread of residuals is roughly constant across all levels of the predictors. If residuals fan out, standard errors and p-values become unreliable.
Normality of residuals: The residuals are approximately normally distributed. This matters most for small samples and for constructing valid confidence intervals.
When you have many candidate predictors, you need a principled way to choose among models:
R-squared always increases (or stays the same) when you add a predictor, so on its own it tends to favour overly complex models.
AIC adds a penalty proportional to the number of parameters. It selects the model that best balances fit and parsimony.
BIC penalises complexity more heavily than AIC and tends to prefer simpler models, especially with larger samples.
Comparing models on these criteria lets you find the set of predictors that explains the data well without overfitting.
F-test for overall model utility: Tests the null hypothesis that all slope coefficients are zero (i.e. none of the predictors matter). A significant F-test means the model as a whole has explanatory power.
t-tests for individual coefficients: Each predictor gets its own t-test. A significant result means that predictor contributes something beyond what the other predictors already explain.
Prediction: Once the model is validated, you can generate confidence intervals (for the mean response) and prediction intervals (for a new individual observation) at any set of predictor values.
MLR equation: Y = β₀ + β₁X₁ + β₂X₂ + ... + βₖXₖ + ε
Log-transformed example (Mesquite Tree Data): log(z) = β₀ + β₁ log(u₁) + β₂ log(u₂) + β₃ log(u₃) + β₄ log(u₄) + ε
Where z = leaf weight, u₁ = canopy diameter (longest axis), u₂ = canopy diameter (shortest axis), u₃ = total height, u₄ = canopy height. Log transformations are used here because the raw relationship between the variables is multiplicative rather than additive.
Multiple linear regression is used in fields ranging from economics (predicting GDP from trade, investment, and labour inputs) to ecology (predicting biomass from environmental measurements, as in the mesquite tree example). Any time a researcher wants to isolate the effect of one factor while controlling for others, MLR is typically the starting point.
Students often think a high R-squared means the model is "correct." A model can explain a lot of variance and still be badly specified, with omitted variables or violated assumptions.
Students often confuse correlation with causation in regression output. A significant coefficient tells you there is an association after controlling for other predictors, not that changing X causes Y to change.
Students sometimes think adding more predictors always improves a model. In practice, adding irrelevant predictors inflates variance and can make the model less useful for prediction (overfitting).
Students frequently interpret each coefficient as if the other variables do not exist. Each βᵢ is the effect of Xᵢ holding the other predictors constant, which can differ substantially from the simple bivariate correlation.
⚠️ Be able to write and interpret the MLR equation, identifying what each symbol represents.
⚠️ Know how to state and check the four OLS assumptions (linearity, independence, homoscedasticity, normality of residuals).
⚠️ Understand the difference between the F-test (whole model) and t-tests (individual coefficients) and when each is used.
⚠️ Be ready to compare models using R², AIC, and BIC, and explain why R² alone is not enough.
⚠️ Know the distinction between a confidence interval for the mean response and a prediction interval for a new observation, and why the prediction interval is wider.
True or False: R-squared can decrease when you add a new predictor to a multiple regression model.
A: False. R² can only stay the same or increase with additional predictors (though adjusted R² can decrease).
Fill in the blank: The method that minimises the sum of squared residuals to estimate regression coefficients is called ________.
A: Ordinary Least Squares (OLS).
True or False: If the F-test for overall model utility is not significant, individual predictors can still have significant t-tests.
A: It is technically possible but unusual; in practice, a non-significant F-test suggests the model as a whole lacks explanatory power.
Fill in the blank: AIC and BIC both penalise models for ________, to guard against overfitting.
A: Having more parameters (model complexity).
Q: Write the general equation for a multiple linear regression model with k predictors and define each term.
A: Y = β₀ + β₁X₁ + β₂X₂ + ... + βₖXₖ + ε. Y is the dependent variable, X₁ through Xₖ are the independent variables, β₀ is the intercept, β₁ through βₖ are the slope coefficients, and ε is the error term.
Q: A researcher fits two models to the same data set. Model A has an AIC of 312 and Model B has an AIC of 298. Which model is preferred and why?
A: Model B is preferred because a lower AIC indicates a better balance between goodness of fit and model complexity.
Q: Explain why a prediction interval is always wider than a confidence interval at the same set of predictor values.
A: A confidence interval estimates the mean response, so it only reflects uncertainty about the population mean. A prediction interval estimates a single new observation, so it includes both the uncertainty about the mean and the additional variability of individual observations around that mean.
Q: In the mesquite tree example, all variables are log-transformed. What does this suggest about the nature of the original relationship?
A: It suggests the relationship between the raw variables is multiplicative (or follows a power-law form) rather than additive. Log-transforming converts a multiplicative relationship into an additive one that linear regression can handle.
Q: What is the consequence of violating the homoscedasticity assumption in OLS regression?
A: The coefficient estimates themselves remain unbiased, but the standard errors become unreliable, leading to incorrect confidence intervals, prediction intervals, and p-values.
This material connects directly to Analysis of Variance (ANOVA), which can be understood as a special case of regression where the predictors are categorical (group indicators). Understanding MLR also lays the groundwork for more advanced methods such as logistic regression (when the outcome is binary) and generalised linear models.
multiple linear regression, MLR, multivariate regression, OLS, ordinary least squares, regression coefficients, beta coefficients, slope, intercept, R-squared, coefficient of determination, adjusted R-squared, AIC, Akaike information criterion, BIC, Bayesian information criterion, F-test, t-test, model selection, overfitting, underfitting, residuals, homoscedasticity, heteroscedasticity, normality, linearity, independence, confidence interval, prediction interval, mesquite tree regression, log transformation, STAT 101, principles of statistics