Difficulty: Intermediate–Advanced | Prerequisites: Parts 1 and 2 of these notes (SLR model, least squares estimation, residuals, R²). Familiarity with t-distributions and confidence intervals from STAT 211.
Having estimated the regression line and checked the assumptions, the next question is: how precise are those estimates, and can we use them to make predictions? This section covers confidence intervals and hypothesis tests for the slope, confidence intervals for the mean response at a given x, prediction intervals for a future observation, and the correlation coefficient. These are the inferential tools that turn a fitted line into actionable conclusions. You need the model, the assumptions, and the residual-checking material from Parts 1 and 2 to follow along.
The slope estimator β̂₁ has a known sampling distribution, which lets you build confidence intervals and run t-tests. You can also construct intervals for the average Y at a particular x (confidence interval) or for a single future Y (prediction interval); the prediction interval is always wider. The correlation coefficient ρ measures the strength of the linear association between two variables, and its square equals R².
Standard error of β̂₁
σ̂_{β̂₁} = σ̂ / √[Σ(x_i − x̄)²]. Estimates how much β̂₁ would vary across repeated samples. Think of it as the ruler for measuring imprecision in the slope estimate.
t-statistic (for β₁)
T = (β̂₁ − β₁₀) / σ̂_{β̂₁}. The standardised distance of the slope estimate from a hypothesised value β₁₀. Follows a t-distribution with n − 2 degrees of freedom.
Confidence interval for β₁
β̂₁ ± t_{n−2, α/2} · σ̂_{β̂₁}. An interval that, with (1 − α)×100% confidence, contains the true slope. In simple terms, it tells you the plausible range for how steeply Y changes with x.
Confidence interval for E(Y) at x = x₀
An interval for the average response across the entire sub-population where x equals x₀. Think of it as: "what is the mean height of all sons whose fathers are 68 inches tall?"
Prediction interval for Y at x = x₀
An interval for a single future observation at x = x₀. Think of it as: "how tall will this particular son be?" It is wider than the confidence interval because it includes both estimation uncertainty and individual random variation.
Correlation coefficient (ρ)
ρ = Cov(X, Y) / (σ_X · σ_Y). A measure of the strength and direction of the linear relationship between two random variables. Ranges from −1 to +1.
Sample correlation coefficient (ρ̂)
Also called Pearson's product-moment correlation. Estimated from data as Σ(X_i − X̄)(Y_i − Ȳ) / [Σ(X_i − X̄)² · Σ(Y_i − Ȳ)²]^{1/2}. In simple terms, it is the data-based estimate of how linearly related two variables are.
Regression to the mean
Galton's observation that sons of unusually tall (or short) fathers tend to be closer to the population average than their fathers were. The slope of the regression line being less than 1 reflects this phenomenon. In simple terms, extreme values in one generation tend to be less extreme in the next.
Under the SLR assumptions, β̂₁ has three key properties:
Unbiased: E(β̂₁) = β₁.
Variance: Var(β̂₁) = σ² / Σ(x_i − x̄)².
Normally distributed (because it is a linear combination of the normally distributed Y_i's).
The proof of unbiasedness works by rewriting the numerator of β̂₁ as Σ(x_i − x̄)Y_i, substituting the model Y_i = β₀ + β₁x_i + ε_i, and noting that Σ(x_i − x̄) = 0. The result is β̂₁ = β₁ + Σε_i(x_i − x̄) / Σ(x_i − x̄)², and taking expectations gives E(β̂₁) = β₁ because E(ε_i) = 0.
Since (β̂₁ − β₁) / σ̂_{β̂₁} follows a t-distribution with n − 2 degrees of freedom, a 100(1 − α)% confidence interval is:
β̂₁ ± t_{n−2, α/2} · σ̂_{β̂₁}
For Galton's data: β̂₁ = 0.514, σ̂_{β̂₁} = 0.027. With n = 1078 (so n − 2 is large, making t_{n−2, 0.025} ≈ 1.96), the 95% CI is 0.514 ± 1.96(0.027) = (0.461, 0.567). We are 95% confident the true slope lies in this range.
In R: confint(lm(y ~ x), level = 0.95).
To test H₀: β₁ = β₁₀, compute:
T = (β̂₁ − β₁₀) / σ̂_{β̂₁}
Compare to the t_{n−2} distribution.
Two-sided H_a (β₁ ≠ β₁₀): reject if |T| > t_{n−2, α/2}.
One-sided H_a (β₁ > β₁₀): reject if T > t_{n−2, α}.
One-sided H_a (β₁ < β₁₀): reject if T < −t_{n−2, α}.
The most important special case is H₀: β₁ = 0, which tests whether x has any linear relationship with Y at all. For Galton's data, T = 19.01 with a p-value below 2 × 10⁻¹⁶, providing overwhelming evidence that father's height and son's height are linearly related.
The estimated mean response at x = x₀ is β̂₀ + β̂₁x₀. Its variance is:
σ² [1/n + (x₀ − x̄)² / Σ(x_i − x̄)²]
A 100(1 − α)% confidence interval for E(Y) at x₀ is:
(β̂₀ + β̂₁x₀) ± t_{n−2, α/2} · SE(x₀)
where SE(x₀) = σ̂ · [1/n + (x₀ − x̄)² / Σ(x_i − x̄)²]^{1/2}.
Notice the variance is smallest when x₀ = x̄ and grows as x₀ moves away from the sample mean. Predictions become less precise at the edges of the data.
For Galton's data with x₀ = 68 (fathers 5'8"): the 95% CI for the average son's height is (68.70, 68.99).
In R: predict(lm(y ~ x), newdata = data.frame(x = 68), interval = "confidence", level = 0.95).
To predict a single future Y at x = x₀, the prediction error has variance:
σ² [1 + 1/n + (x₀ − x̄)² / Σ(x_i − x̄)²]
The extra "1" accounts for the individual ε of the new observation. A 100(1 − α)% prediction interval is:
(β̂₀ + β̂₁x₀) ± t_{n−2, α/2} · SE_pred(x₀)
where SE_pred(x₀) = σ̂ · [1 + 1/n + (x₀ − x̄)² / Σ(x_i − x̄)²]^{1/2}.
For Galton's data with x₀ = 72 (Zach's father is 6 feet tall): the 98% prediction interval for Zach's height is (65.22, 76.59). That is a span of over 11 inches, reflecting substantial individual variation.
In R: predict(lm(y ~ x), newdata = data.frame(x = 72), interval = "prediction", level = 0.98).
The prediction interval is always wider than the confidence interval at the same x₀ and confidence level. The confidence interval targets the mean of a sub-population (averaging reduces uncertainty). The prediction interval targets a single draw from that sub-population (the individual's ε adds variability on top of the estimation error).
When neither variable is naturally the "predictor," the correlation coefficient ρ measures how strongly X and Y are linearly related.
Population correlation:
ρ = Cov(X, Y) / (σ_X · σ_Y)
Sample correlation (Pearson's):
ρ̂ = Σ(X_i − X̄)(Y_i − Ȳ) / [Σ(X_i − X̄)² · Σ(Y_i − Ȳ)²]^{1/2}
Key facts:
−1 ≤ ρ̂ ≤ 1.
ρ̂² = R² (in simple linear regression).
β̂₁ = ρ̂ · (σ̂_Y / σ̂_X).
ρ = 0 does not imply independence. Two variables can have a strong non-linear association and still have zero correlation.
The test statistic is:
T = ρ̂√(n − 2) / √(1 − ρ̂²)
Under H₀ with large n, T is approximately N(0, 1). Reject in the usual way for one-sided or two-sided alternatives.
In R: cor.test(x, y) provides the sample correlation, a 95% CI for ρ, and a two-sided test of H₀: ρ = 0.
Standard error of β̂₁: σ̂_{β̂₁} = σ̂ / √[Σ(x_i − x̄)²]
CI for β₁: β̂₁ ± t_{n−2, α/2} · σ̂_{β̂₁}
Test statistic for β₁: T = (β̂₁ − β₁₀) / σ̂_{β̂₁}
Variance of estimated mean at x₀: σ² [1/n + (x₀ − x̄)² / Σ(x_i − x̄)²]
Variance of prediction error at x₀: σ² [1 + 1/n + (x₀ − x̄)² / Σ(x_i − x̄)²]
Sample correlation: ρ̂ = Σ(X_i − X̄)(Y_i − Ȳ) / [Σ(X_i − X̄)² · Σ(Y_i − Ȳ)²]^{1/2}
Test statistic for ρ: T = ρ̂√(n − 2) / √(1 − ρ̂²)
Prediction intervals matter whenever you need to forecast individual outcomes rather than averages. A hospital predicting one patient's recovery time (not the average across all similar patients) needs the wider prediction interval. Correlation is used routinely in finance to measure how closely two asset prices move together, which drives portfolio diversification decisions.
Students often confuse confidence intervals for the mean response with prediction intervals for a future observation. The confidence interval answers "what is the average Y here?" while the prediction interval answers "what will the next individual Y be?" The prediction interval is always wider.
A common error is using n − 1 degrees of freedom instead of n − 2 for the t-statistic in regression. The regression t-test uses n − 2 because two parameters (β₀ and β₁) have been estimated.
Students sometimes believe ρ = 0 means X and Y are unrelated. It means there is no linear relationship. A strong curved relationship can produce ρ = 0.
Interpreting a confidence interval as "95% probability the true parameter is in this interval" is technically incorrect. The parameter is fixed; the interval is random. In repeated sampling, 95% of such intervals would contain the true value.
⚠️ Know the difference between a confidence interval for E(Y|x₀) and a prediction interval for a future Y at x₀. You will likely be asked to compute or interpret both.
⚠️ Testing H₀: β₁ = 0 is the single most commonly tested hypothesis in simple linear regression. Know the formula, the degrees of freedom (n − 2), and how to interpret the result.
⚠️ Be able to state the three properties of β̂₁ (unbiased, known variance, normally distributed).
⚠️ Know that ρ̂² = R² and that ρ = 0 does not imply independence.
⚠️ Understand why prediction intervals widen as x₀ moves away from x̄.
True or false: The prediction interval is always wider than the confidence interval at the same x₀ and confidence level. (True.)
Fill in the blank: The degrees of freedom for the t-test of β₁ in SLR is ____. (n − 2)
True or false: ρ = 0 guarantees that X and Y are independent. (False – it only rules out linear association.)
Fill in the blank: ρ̂² = ____. (R², the coefficient of determination.)
True or false: The variance of β̂₀ + β̂₁x₀ is smallest when x₀ equals x̄. (True.)
Q: For Galton's data, the 95% CI for the average son's height when the father is 68 inches tall is (68.70, 68.99). Interpret this.
A: We are 95% confident that the true average height of all sons whose fathers are 68 inches tall is between 68.70 and 68.99 inches.
Q: The 98% prediction interval for Zach's height (father 72 inches) is (65.22, 76.59). Why is it so much wider than the CI above?
A: The prediction interval accounts for two sources of uncertainty: imprecision in estimating the regression line, and the individual random variation (ε) of Zach's own height around the population mean at x = 72. The confidence interval only accounts for the first source.
Q: State the null hypothesis and test statistic for testing whether the slope is zero.
A: H₀: β₁ = 0. The test statistic is T = β̂₁ / σ̂_{β̂₁}, which follows a t-distribution with n − 2 degrees of freedom under H₀.
Q: For Galton's data, T = 19.01 for testing H₀: β₁ = 0. What do you conclude?
A: With a p-value far below any standard significance level, we reject H₀. There is overwhelming evidence that father's height and son's height are linearly related.
Q: Explain why ρ = 0 does not imply X and Y are independent.
A: Correlation measures only the strength of a linear relationship. Two variables can have a strong non-linear association (e.g. Y = X²) while having ρ = 0 because the positive and negative deviations cancel out.
Q: Write the R command to get a 95% confidence interval for the mean response at x = 68.
A: predict(lm(y ~ x), newdata = data.frame(x = 68), interval = "confidence", level = 0.95)
The t-test for β₁ = 0 is equivalent to the F-test in one-predictor ANOVA, which you will encounter later. Prediction intervals extend naturally to multiple regression, where x₀ becomes a vector. The correlation coefficient connects to the broader topic of association measures and appears again in discussions of multicollinearity. The distinction between estimating a mean response and predicting a single observation carries through to every regression model in the course.
inference, hypothesis test, t-test, slope test, confidence interval, prediction interval, standard error, degrees of freedom, t-distribution, mean response, future observation, Pearson correlation, sample correlation, population correlation, rho, R-squared, regression to the mean, Galton, cor.test, confint, predict, STAT 212, Principles of Statistics II, Texas A&M