Difficulty: Intermediate | Prerequisites: Parts 1 and 2 of these notes (SLR model, least squares, residuals, R²)
Source: Lecture Notes, Chapter 1 (Part A) – Sharmistha Guha, Texas A&M University
Tags: inference, t-test, confidence interval, prediction interval, hypothesis test, slope test, standard error, t-distribution, correlation, Pearson correlation, sample correlation coefficient, beta-1 test, p-value, STAT 212
Fitting a regression line and computing R² tells you what happened in the sample. Inference is the step that connects those sample results to the broader population. This section covers confidence intervals and hypothesis tests for the slope β₁, confidence intervals for the mean response at a given x, prediction intervals for a single future observation, and the closely related topic of correlation. These inferential tools rely on the normality assumptions from Part 1; if those assumptions fail, the p-values and intervals may not be trustworthy.
The slope estimator β̂₁ follows a normal distribution, and once you plug in σ̂ for the unknown σ, the resulting standardised quantity follows a t-distribution with n − 2 degrees of freedom. This gives you confidence intervals and hypothesis tests for β₁. Similar logic yields intervals for the mean response at a particular x (narrower) and prediction intervals for a single new observation (wider). The sample correlation coefficient R is a scale-free measure of linear association and is connected to R² by R² = (sample correlation)².
Standard error of β̂₁
σ̂ divided by the square root of Σ(xᵢ − x̄)². Written σ̂_{β̂₁}. It estimates how much β̂₁ would vary across repeated samples.
t-statistic (for β₁)
T = (β̂₁ − β₁₀) / σ̂_{β̂₁}, where β₁₀ is the hypothesised value of β₁ under H₀. Under the null, T follows a t-distribution with n − 2 degrees of freedom.
Confidence interval for β₁
β̂₁ ± t_{n−2; α/2} · σ̂_{β̂₁}. A (1 − α)100% interval. It tells you the plausible range for the true slope given your data.
Confidence interval for the mean response E(Y) at x = x₀
Centred at β̂₀ + β̂₁x₀ with half-width t_{n−2; α/2} · SE(x₀), where SE(x₀) = σ̂ · √[1/n + (x₀ − x̄)² / Σ(xᵢ − x̄)²]. This interval targets the average Y in the sub-population where x equals x₀.
Prediction interval for a new Y at x = x₀
Centred at β̂₀ + β̂₁x₀ with half-width t_{n−2; α/2} · SE_pred(x₀), where SE_pred(x₀) = σ̂ · √[1 + 1/n + (x₀ − x̄)² / Σ(xᵢ − x̄)²]. The extra "1" under the square root accounts for the randomness of the individual observation itself. Think of it as the interval for one specific future data point, not the population average.
Population correlation coefficient (ρ)
ρ = Cov(X, Y) / (σ_X · σ_Y). It measures the strength and direction of the linear relationship between two random variables. Ranges from −1 to +1.
Sample correlation coefficient (R)
Also called Pearson's product-moment correlation coefficient: R = Σ(Xᵢ − X̄)(Yᵢ − Ȳ) / √[Σ(Xᵢ − X̄)² · Σ(Yᵢ − Ȳ)²]. Its square equals the coefficient of determination R².
Three key properties drive all the inference:
Unbiased: E(β̂₁) = β₁. The proof works by substituting the model Yᵢ = β₀ + β₁xᵢ + εᵢ into the formula for β̂₁ and showing that the error terms wash out in expectation (because E(εᵢ) = 0).
Variance: Var(β̂₁) = σ² / Σ(xᵢ − x̄)². More spread-out x-values (a larger denominator) produce a more precise estimate of the slope.
Normality: β̂₁ is a linear combination of the Yᵢ, which are themselves normal, so β̂₁ is normally distributed.
Replacing σ with its estimate σ̂ in the standardised version of β̂₁ gives:
T = (β̂₁ − β₁) / σ̂_{β̂₁}
This follows a t-distribution with n − 2 degrees of freedom (not n − 1, because two parameters were estimated to build σ̂).
A (1 − α)100% confidence interval is:
β̂₁ ± t_{n−2; α/2} · σ̂_{β̂₁}
To test H₀: β₁ = β₁₀, compute T = (β̂₁ − β₁₀) / σ̂_{β̂₁} and compare to the t_{n−2} distribution.
Two-sided alternative (β₁ ≠ β₁₀): reject if |T| > t_{n−2; α/2}.
One-sided alternative (β₁ > β₁₀): reject if T > t_{n−2; α}.
One-sided alternative (β₁ < β₁₀): reject if T < −t_{n−2; α}.
The most common test is H₀: β₁ = 0, which asks whether there is any linear relationship between x and Y at all. If β₁ = 0, the predictor x contributes nothing to explaining Y.
For the Galton data, β̂₁ = 0.514 with σ̂_{β̂₁} = 0.027. Testing H₀: β₁ = 0 gives T = 19.01, with a p-value below 2 × 10⁻¹⁶. There is overwhelming evidence that fathers' and sons' heights are positively linearly related.
A 95% confidence interval for β₁:
0.514 ± 1.96 × 0.027 = (0.461, 0.567)
We are 95% confident the true slope falls between 0.461 and 0.567.
Sometimes you want to estimate the average response across the whole sub-population at a specific x-value, not the slope itself.
The estimator is β̂₀ + β̂₁x₀, and a (1 − α)100% confidence interval is:
(β̂₀ + β̂₁x₀) ± t_{n−2; α/2} · SE(x₀)
where SE(x₀) = σ̂ · √[1/n + (x₀ − x̄)² / Σ(xᵢ − x̄)²].
Notice that the interval is narrowest when x₀ = x̄ and widens as x₀ moves away from the centre of the data.
For the Galton data with x₀ = 68 (fathers 5′8″), a 95% confidence interval for the average son's height is (68.70, 68.99).
Predicting one specific future observation is harder than estimating the mean, because the individual observation also has its own εᵢ noise on top of the estimation uncertainty.
A (1 − α)100% prediction interval is:
(β̂₀ + β̂₁x₀) ± t_{n−2; α/2} · SE_pred(x₀)
where SE_pred(x₀) = σ̂ · √[1 + 1/n + (x₀ − x̄)² / Σ(xᵢ − x̄)²].
The "1 +" inside the square root is the critical difference. It ensures the prediction interval is always wider than the corresponding confidence interval for the mean, which makes intuitive sense: predicting where one data point will land is inherently less certain than estimating the average.
For the Galton data with x₀ = 72 (father is six feet tall), a 98% prediction interval for Zach's height is (65.22, 76.59).
# Confidence interval for E(Y) at x = 68
predict(lm(y ~ x), newdata = data.frame(x = 68),
interval = "confidence", level = 0.95)
# Prediction interval for Y at x = 72
predict(lm(y ~ x), newdata = data.frame(x = 72),
interval = "prediction", level = 0.98)
# Confidence interval for slope
confint(lm(y ~ x), level = 0.95)
Both intervals are centred at the same point, β̂₀ + β̂₁x₀. The confidence interval targets the population mean; the prediction interval targets a single observation. The prediction interval is always wider because it adds the irreducible noise σ² to the estimation uncertainty.
Sometimes you want to measure the strength of the linear association without designating one variable as predictor and the other as response.
The population correlation is ρ = Cov(X, Y) / (σ_X σ_Y), and the sample estimate is:
R = Σ(Xᵢ − X̄)(Yᵢ − Ȳ) / √[Σ(Xᵢ − X̄)² · Σ(Yᵢ − Ȳ)²]
Key facts:
−1 ≤ R ≤ 1. The closer |R| is to 1, the stronger the linear relationship.
R² equals the coefficient of determination from the regression.
The slope estimator can be written as β̂₁ = R · (σ̂_Y / σ̂_X).
The test statistic is T = R√(n − 2) / √(1 − R²). For large n, T is approximately standard normal under H₀. In R, cor.test(x, y) provides R, a 95% confidence interval for ρ, and a two-sided test of H₀: ρ = 0.
An important caveat: ρ = 0 does not imply X and Y are independent. It only rules out a linear relationship. A strong nonlinear association (e.g. a parabola) can have ρ close to zero.
Quantity | Formula |
|---|---|
Standard error of β̂₁ | σ̂_{β̂₁} = σ̂ / √[Σ(xᵢ − x̄)²] |
t-statistic for slope | T = (β̂₁ − β₁₀) / σ̂_{β̂₁} |
CI for β₁ | β̂₁ ± t_{n−2; α/2} · σ̂_{β̂₁} |
SE for mean response | SE(x₀) = σ̂ √[1/n + (x₀ − x̄)²/Σ(xᵢ − x̄)²] |
SE for prediction | SE_pred(x₀) = σ̂ √[1 + 1/n + (x₀ − x̄)²/Σ(xᵢ − x̄)²] |
Sample correlation | R = Σ(Xᵢ − X̄)(Yᵢ − Ȳ) / √[Σ(Xᵢ − X̄)² Σ(Yᵢ − Ȳ)²] |
t-statistic for ρ = 0 | T = R√(n − 2) / √(1 − R²) |
Prediction intervals are widely used in clinical settings (predicting a patient's cholesterol at a future visit given their age) and in manufacturing (predicting the tensile strength of the next batch given process temperature). Correlation is the standard first-pass measure of association in virtually every empirical discipline, from finance (returns of two assets) to psychology (scores on two tests).
Students regularly confuse the confidence interval for E(Y) with the prediction interval for a single Y. The confidence interval estimates the population mean at x₀; the prediction interval captures where a single new observation is likely to fall. The prediction interval is always wider.
Testing H₀: β₁ = 0 with a two-sided alternative is mathematically equivalent to testing H₀: ρ = 0. Some students treat these as separate tests when they are, in simple linear regression, the same.
Students sometimes claim that ρ = 0 implies independence. That is only true if the joint distribution of (X, Y) is bivariate normal. For general distributions, zero correlation means no linear relationship, but a nonlinear dependence may still exist.
The degrees of freedom for the t-statistic in regression are n − 2, not n − 1. This trips students up, especially those coming directly from one-sample t-test material.
⚠️ Know the formula for σ̂_{β̂₁} and be able to construct a confidence interval or carry out a t-test for β₁.
⚠️ Be able to state why prediction intervals are wider than confidence intervals for the mean, and identify which formula has the extra "1" under the root.
⚠️ The test of H₀: β₁ = 0 is the most commonly tested inferential concept in this chapter. Know its test statistic, distribution, and interpretation.
⚠️ Understand the relationship between R and R²: the square of the sample correlation coefficient equals the coefficient of determination.
⚠️ Remember that ρ = 0 does not imply independence (unless the distribution is bivariate normal).
True or False: The t-statistic for testing β₁ = 0 uses n − 1 degrees of freedom.
Fill in the blank: A prediction interval for Y at x₀ is always ________ than the corresponding confidence interval for E(Y) at x₀.
True or False: R² = 0.251 implies R = 0.251.
Fill in the blank: E(β̂₁) = β₁, which means β̂₁ is a(n) ________ estimator.
True or False: If ρ = 0, X and Y must be independent.
Answers: 1. False (n − 2). 2. Wider. 3. False (R = √0.251 ≈ 0.501, assuming a positive association). 4. Unbiased. 5. False (zero correlation only implies independence under bivariate normality; otherwise, nonlinear dependence can still exist).
Q: For the Galton data, β̂₁ = 0.514, σ̂_{β̂₁} = 0.027, and n = 1078. Test H₀: β₁ = 0 at the 5% significance level.
A: T = 0.514 / 0.027 = 19.01. With n − 2 = 1076 degrees of freedom, the critical value t_{1076; 0.025} ≈ 1.96. Since 19.01 >> 1.96, we reject H₀. There is very strong evidence that the slope is not zero, i.e. fathers' and sons' heights are linearly related.
Q: Write the formula for a 95% prediction interval for Y when x = x₀. How does it differ from the confidence interval for E(Y)?
A: Prediction interval: (β̂₀ + β̂₁x₀) ± t_{n−2; 0.025} · σ̂√[1 + 1/n + (x₀ − x̄)²/Σ(xᵢ − x̄)²]. The confidence interval for E(Y) uses the same centre but replaces the "1 + 1/n + ..." under the root with just "1/n + ...". The extra 1 in the prediction formula accounts for the noise in the individual observation.
Q: Explain the relationship between the sample correlation R and the coefficient of determination R².
A: R² = (sample correlation)². In simple linear regression, R² is the fraction of variation in Y explained by x, and R is the signed measure of linear association. The slope estimate can also be written β̂₁ = R · (σ̂_Y / σ̂_X).
Q: A 95% confidence interval for the mean son's height when the father is 5′8″ is (68.70, 68.99). A 98% prediction interval for Zach's height when his father is 6′0″ is (65.22, 76.59). Why is the prediction interval so much wider?
A: Two reasons. First, the prediction interval includes an extra source of variability: the individual noise ε for Zach's particular height, on top of the uncertainty in estimating β₀ and β₁. Second, x₀ = 72 is further from x̄ than x₀ = 68, which inflates the (x₀ − x̄)² term. The dominant factor, though, is the extra "1" in the SE formula, which reflects irreducible individual-level randomness.
Q: State the test statistic for H₀: ρ = 0 and its approximate distribution.
A: T = R√(n − 2) / √(1 − R²). Under H₀ and for large n, T is approximately N(0, 1). For smaller n, it follows a t-distribution with n − 2 degrees of freedom.
Inference for β₁ directly extends to each coefficient in multiple regression, where each predictor gets its own t-test and confidence interval. Prediction intervals reappear in the same form (with a more complex variance expression) in the multiple regression setting. Correlation connects to the broader topic of covariance matrices, which underpin multivariate statistics, principal component analysis, and factor analysis later in many statistics curricula.
inference for slope, t-test regression, confidence interval beta-1, prediction interval, standard error of slope, t-distribution, degrees of freedom n minus 2, hypothesis test beta equals zero, Pearson correlation, sample correlation coefficient, population correlation, rho, cor.test, confint, predict function R, mean response interval, new observation interval, STAT 212, Principles of Statistics II