Source: Lecture Notes, Chapter 1 (Part C) – Dr. Abhishek Chakrabortty, Texas A&M University
Tags: variance estimation, MSE, SSE, SSR, SST, R-squared, coefficient of determination, t-test regression coefficient, F-test model utility, reduction method, partial F-test, nested models, confidence interval regression, prediction interval regression, STAT 212
Difficulty: Intermediate Prerequisites: Part 1 of these notes (Model Setup and Least Squares Estimation), t-distribution, F-distribution basics.
Once you have fitted a multiple regression model and obtained coefficient estimates, the natural next questions are: How well does the model fit? Are specific predictors contributing anything? Could you drop some predictors without losing much? This section covers the machinery for answering those questions, including variance estimation, the coefficient of determination (R²), individual t-tests for single coefficients, the overall F-test for model utility, and the reduction method for testing whether a subset of predictors can be removed. If Part 1 was about building the model, this part is about interrogating it.
We estimate σ² using MSE = SSE/(n − k − 1), measure overall fit with R² = SSR/SST, test individual coefficients with t-tests, and test the whole model (or subsets of predictors) with F-tests. The reduction method compares a full model against a reduced model to decide whether dropped predictors were needed.
Fitted / predicted values (ŷᵢ)
ŷᵢ = β̂₀ + β̂₁xᵢ₁ + β̂₂xᵢ₂ + ... + β̂ₖxᵢₖ. The model's estimate of each observed yᵢ. Think of it as "what the model predicts for observation i."
Sum of Squared Errors (SSE)
SSE = Σᵢ₌₁ⁿ (yᵢ − ŷᵢ)². Measures the total residual variation left unexplained by the model.
Sum of Squares for Regression (SSR)
SSR = Σᵢ₌₁ⁿ (ŷᵢ − ȳ)². Measures how much of the total variation in Y is explained by the model.
Total Sum of Squares (SST)
SST = Σᵢ₌₁ⁿ (yᵢ − ȳ)². The total variation in Y around its mean. The fundamental decomposition is SST = SSR + SSE.
Mean Squared Error (MSE)
MSE = SSE / (n − k − 1). The unbiased estimate of σ², the error variance. The denominator n − k − 1 accounts for the k + 1 parameters estimated.
Coefficient of determination (R²)
R² = SSR / SST. The fraction of total variation in Y explained by the model. In simple terms, an R² of 0.88 means the model accounts for about 88% of the variability in the response.
Variance of a coefficient estimate
Var(β̂ᵢ) = σ²cᵢ₊₁, where cⱼ is the jth diagonal element of (XᵀX)⁻¹. In simple terms, each coefficient's precision depends on the error variance scaled by a factor from the geometry of the predictor data.
t-test for a single regression coefficient
Tests H₀: βᵢ = 0 (predictor xᵢ is not needed, given the other predictors in the model). The test statistic is t = β̂ᵢ / SE(β̂ᵢ), compared to a t-distribution with n − k − 1 degrees of freedom. This is the p-value shown next to each coefficient in R's summary(lm(...)) output.
F-test for model utility
Tests H₀: β₁ = β₂ = ... = βₖ = 0 (none of the predictors are related to Y). In simple terms, this asks whether the entire model is statistically significant.
F-distribution
A right-skewed distribution used for ratio-based test statistics. Under H₀ for the model utility test, the test statistic follows an F-distribution with k and n − k − 1 degrees of freedom.
Reduction method (partial F-test)
A technique for testing whether a specific subset of predictors can be dropped from the model. It compares SSE from a full model (all predictors) to SSE from a reduced model (subset removed), using an F-statistic. Think of it as asking: "Does the increase in SSE from dropping these variables exceed what we would expect by chance?"
Full model
The model containing all k independent variables. Its SSE is denoted SSE_f.
Reduced model
The model with ℓ predictors removed (containing only k − ℓ predictors). Its SSE is denoted SSE_r. By definition, SSE_r ≥ SSE_f, because the full model has more flexibility to fit the data.
After fitting the model:
σ̂² = SSE / (n − k − 1) = MSE
The residual standard error reported by R is σ̂ = √MSE.
SST = SSR + SSE
R² = SSR / SST
R² always lies between 0 and 1. Higher values indicate a better fit. In the mesquite tree example, R² = 0.884, meaning the four log-transformed predictors explain about 88% of the variation in log(leaf weight).
For a given predictor vector xᵀ = [1, x₁, x₂, ..., xₖ]:
Point estimate of E(Y): μ̂(x) = β̂₀ + β̂₁x₁ + ... + β̂ₖxₖ
Confidence interval for E(Y) at X = x:
μ̂(x) ± t_(n−k−1, α/2) · σ̂ · √(xᵀ(XᵀX)⁻¹x)
Prediction interval for a new Y at X = x:
μ̂(x) ± t_(n−k−1, α/2) · σ̂ · √(1 + xᵀ(XᵀX)⁻¹x)
The prediction interval is always wider than the confidence interval because it accounts for both the uncertainty in estimating the mean and the inherent variability of individual observations.
In R:
predict(fit, X, interval="conf") for the confidence interval
predict(fit, X, interval="predict") for the prediction interval
This tests whether the model as a whole is useful:
H₀: β₁ = β₂ = ... = βₖ = 0 (Y is unrelated to all predictors)
Hₐ: at least one βⱼ ≠ 0
The test statistic:
F = (SSR/k) / MSE = (R²/k) / ((1 − R²)/(n − k − 1))
Under H₀ (with normally distributed errors), F follows an F-distribution with degrees of freedom k and n − k − 1.
Reject H₀ if F ≥ F_(k, n−k−1; α), or equivalently if the p-value is less than α.
In the mesquite tree example: F = 28.6, with p-value = 7.307 × 10⁻⁷. The model is highly significant.
To test H₀: βᵢ = 0 (predictor xᵢ is not needed given the others), use the t-statistic and p-value from the summary(lm(...)) output.
In the mesquite example, for x₂ (log-diam2): t = 2.314, p = 0.0353. At α = 0.05, we reject H₀ and conclude that x₂ should remain in the model.
For x₁ (log-diam1): t = 1.490, p = 0.1569. Not significant at α = 0.05, suggesting x₁ may not be needed once the other predictors are included.
When you want to test whether multiple predictors can be dropped simultaneously, do not run separate t-tests for each one. Separate tests can lead to incorrect conclusions because they do not account for the joint effect of removing several variables at once.
Instead, use the reduction method.
Setup: Suppose the full model has k predictors and you want to test whether the last ℓ of them have zero coefficients:
H₀: β_(k−ℓ+1) = β_(k−ℓ+2) = ... = βₖ = 0
Steps:
Fit the full model (all k predictors). Record SSE_f.
Fit the reduced model (only the first k − ℓ predictors). Record SSE_r.
Compute the F-statistic:
F = ((SSE_r − SSE_f) / ℓ) / (SSE_f / (n − k − 1))
Compare F to the F-distribution with ℓ and n − k − 1 degrees of freedom.
Reject H₀ if F ≥ F_(ℓ, n−k−1; α).
Why SSE_r ≥ SSE_f: The full model includes every predictor available to minimise SSE. The reduced model has fewer predictors, so it cannot fit the data as closely. The only way SSE_r = SSE_f is if the removed predictors contributed nothing at all to reducing the error.
Hypothesis: H₀: β₁ = 0, β₃ = 0 (we do not need log-diam1 and log-totht alongside log-diam2 and log-canht).
Full model (x₁, x₂, x₃, x₄): SSE_f = 2.0368
Reduced model (x₂, x₄ only): SSE_r = 2.8261
ℓ = 2 predictors dropped, n = 20, k = 4
F = ((2.8261 − 2.0368) / 2) / (2.0368 / 15) = 2.906
The critical value F_(2, 15; 0.05) = 3.68. Since 2.906 < 3.68, we do not reject H₀. There is not enough evidence to say we need x₁ and x₃ in the model alongside x₂ and x₄.
The reduced model's R² is 0.84, compared with 0.88 for the full model. The simpler model explains nearly as much variance, and it requires only two measurements per tree instead of four.
Item | Formula |
|---|---|
Fitted values | ŷᵢ = β̂₀ + β̂₁xᵢ₁ + ... + β̂ₖxᵢₖ |
SSE | Σ(yᵢ − ŷᵢ)² |
SSR | Σ(ŷᵢ − ȳ)² |
SST | Σ(yᵢ − ȳ)² = SSR + SSE |
MSE | SSE / (n − k − 1) |
R² | SSR / SST |
F-stat (model utility) | (SSR/k) / MSE = (R²/k) / ((1 − R²)/(n − k − 1)) |
F-stat (reduction) | ((SSE_r − SSE_f) / ℓ) / (SSE_f / (n − k − 1)) |
CI for E(Y) | μ̂(x) ± t · σ̂ · √(xᵀ(XᵀX)⁻¹x) |
PI for Y | μ̂(x) ± t · σ̂ · √(1 + xᵀ(XᵀX)⁻¹x) |
The reduction method is used whenever a researcher wants to simplify a model. In clinical trials, you might test whether demographic variables (age, sex) add predictive power beyond the treatment indicator. In economics, you might test whether regional dummy variables improve a wage regression that already includes education and experience. The principle is always the same: compare a full model to a nested, simpler one.
Students often test multiple individual coefficients one at a time (each at α = 0.05) and combine the results to conclude whether a group of predictors is needed. This inflates the overall Type I error rate and can give misleading answers. Use the reduction method (partial F-test) instead.
A high R² does not mean every predictor in the model is important. Some predictors may be redundant because they are correlated with others already in the model.
Students sometimes confuse the confidence interval for E(Y) with the prediction interval for a new Y. The prediction interval is always wider because it includes the variability of individual observations around the mean.
The F-test for model utility tells you that at least one predictor matters. It does not tell you which one(s). You need individual t-tests or the reduction method for that.
⚠️ Be able to compute and interpret R² from SSR and SST.
⚠️ Know the F-test for model utility: hypotheses, test statistic, degrees of freedom, rejection rule.
⚠️ Know the reduction method: how to set up H₀, compute the F-statistic from SSE_f and SSE_r, and state the conclusion.
⚠️ Understand why SSE_r ≥ SSE_f (the full model has more flexibility).
⚠️ Know the difference between a confidence interval for E(Y) and a prediction interval for Y, and which R commands produce each.
⚠️ Be able to read R output from summary(lm(...)) and identify coefficient estimates, standard errors, t-values, p-values, R², and the F-statistic.
True or False: SST = SSR + SSE.
Fill in the blank: MSE is computed as SSE divided by _______.
True or False: To test whether three predictors can simultaneously be dropped from a model, you should run three separate t-tests.
Fill in the blank: The F-test for model utility has _______ and _______ degrees of freedom.
True or False: The prediction interval for Y at a given x is narrower than the confidence interval for E(Y) at that same x.
Answers: 1. True. 2. n − k − 1. 3. False (use the reduction method / partial F-test). 4. k and n − k − 1. 5. False (the prediction interval is wider).
Q: In the mesquite tree regression, the F-statistic for model utility is 28.6 with a p-value of 7.307 × 10⁻⁷. What do you conclude at α = 0.05?
A: We reject H₀: β₁ = β₂ = β₃ = β₄ = 0. At least one of the four predictors has a non-zero coefficient, so the model is useful.
Q: Explain why you should not test H₀: β₁ = 0 and H₀: β₃ = 0 separately when your real question is whether both β₁ and β₃ are zero.
A: Separate tests each control their own Type I error rate, but the combined probability of making at least one error increases. More importantly, the individual t-tests each condition on the other predictors being in the model. They do not capture the joint effect of removing both variables simultaneously. The reduction method handles this correctly by comparing full and reduced models in a single test.
Q: In the mesquite example, the reduction method gives F = 2.906 with a critical value of 3.68 at α = 0.05. State the conclusion and its practical implication.
A: We fail to reject H₀: β₁ = 0, β₃ = 0. There is insufficient evidence that log-diam1 and log-totht are needed alongside log-diam2 and log-canht. The practical implication is that we can use a simpler two-predictor model, requiring only two measurements per tree rather than four.
Q: A model has R² = 0.884 and n = 20, k = 4. Compute the F-statistic for model utility.
A: F = (R²/k) / ((1 − R²)/(n − k − 1)) = (0.884/4) / ((1 − 0.884)/(20 − 4 − 1)) = (0.221) / (0.116/15) = 0.221 / 0.00773 ≈ 28.6.
Q: What R commands produce (a) a confidence interval and (b) a prediction interval for a fitted model object called fit at new data stored in X?
A: (a) predict(fit, X, interval="conf") and (b) predict(fit, X, interval="predict").
The F-test for model utility is a generalisation of the t-test for the slope in simple linear regression (with k = 1, the F-test and the two-sided t-test are equivalent: F = t²). The reduction method connects to ANOVA, where you are effectively comparing nested models. The ideas here feed directly into model selection (covered in Part 3 of these notes), where AIC and BIC formalise the trade-off between fit and complexity.
Related Terms / Search Tags: F-test, partial F-test, reduction method, model utility test, R-squared, coefficient of determination, SSE, SSR, SST, MSE, variance estimation, confidence interval regression, prediction interval regression, nested models, t-test regression, STAT 212, ANOVA regression, degrees of freedom, mesquite tree data