Difficulty: Intermediate | Prerequisites: Simple linear regression model, least squares estimation, and model assumptions (see Part 1 notes).
Once you have fitted a regression line, the natural next questions are: how well does it fit, is the relationship statistically significant, and how can you use it to make predictions? This set of notes covers the tools for answering all three. Residual analysis checks whether the model assumptions hold. R² tells you how much of the variation in y is explained by x. Hypothesis testing and confidence intervals let you make formal inferences. Correlation analysis measures the strength of the linear relationship from a different angle. Together, these topics complete the simple linear regression toolkit.
Residuals are the gaps between what you observed and what the line predicted; plotting them reveals assumption violations. R² measures how much of the total variation in y is captured by the model. Hypothesis tests on the slope tell you whether the relationship is statistically significant. Confidence intervals estimate the mean response, while prediction intervals estimate individual future observations (and are always wider). The Pearson correlation coefficient r summarises the strength and direction of the linear relationship between two variables.
Residual (ei)
The difference between an observed value and its fitted value: ei = yi − ŷi. In simple terms, it is how far each data point sits from the regression line.
Coefficient of determination (R²)
The proportion of total variation in y that is explained by the linear model. Think of it as a percentage: an R² of 0.75 means 75% of the variability in y is accounted for by x.
Total Sum of Squares (SST)
Σ(yi − ȳ)². The total variability in y around its mean, before any model is fitted.
Regression Sum of Squares (SSR)
Σ(ŷi − ȳ)². The portion of total variability that the regression line explains.
Error Sum of Squares (SSE)
Σ(yi − ŷi)². The portion of total variability left unexplained, i.e. the sum of squared residuals.
Standard error of the slope, SE(β̂1)
A measure of how precisely the slope has been estimated. Smaller values mean more certainty about the true slope.
t-statistic (for the slope)
The test statistic T = β̂1 / SE(β̂1), used to test whether the true slope differs from zero. It follows a t-distribution with n − 2 degrees of freedom.
p-value
The probability of observing a test statistic as extreme as (or more extreme than) the one calculated, assuming the null hypothesis is true. A small p-value provides evidence against H0.
Confidence interval for the mean response
An interval estimate for E(Y | x0), the average value of y at a particular x0. It captures uncertainty about where the regression line sits.
Prediction interval for a future observation
An interval estimate for an individual future y at a particular x0. It is wider than the confidence interval because it accounts for both estimation error and the natural scatter of individual observations around the line.
Pearson correlation coefficient (r or R)
A number between −1 and +1 that measures the strength and direction of the linear relationship between two variables. In simple terms, +1 is a perfect upward line, −1 is a perfect downward line, and 0 means no linear pattern.
Population correlation coefficient (ρ)
The true correlation between X and Y in the population. The sample r is an estimate of ρ.
Covariance, Cov(X, Y)
A measure of how two variables move together. Positive covariance means they tend to increase together; negative means one tends to decrease as the other increases.
A residual for observation i is:
ei = yi − ŷi
where ŷi = β̂0 + β̂1·xi.
Residual plots are the primary diagnostic tool. Three plots to know:
Residuals vs. x (or vs. fitted values): If the assumptions hold, the plot should show a random scatter with no pattern. A curve suggests non-linearity. A funnel shape (wider scatter on one side) suggests heteroscedasticity.
Residuals vs. fitted values (ŷ): Functionally similar to residuals vs. x in simple regression. A widening or narrowing fan indicates the constant-variance assumption is violated.
Normal probability plot (Q-Q plot) of residuals: Points should fall roughly along a straight diagonal line. Systematic departures (S-shapes, heavy tails) indicate the normality assumption may not hold.
Histograms and density plots of residuals serve the same purpose as Q-Q plots but are less precise for small samples.
R² decomposes total variation into explained and unexplained parts:
R² = SSR / SST = 1 − SSE / SST
Key relationships:
SST = SSR + SSE (the total variation splits cleanly into model and error).
R² ranges from 0 to 1.
An R² of 0 means the model explains none of the variation; an R² of 1 means it explains all of it.
In simple linear regression, R² equals the square of the Pearson correlation coefficient: R² = r².
An R² of 0.25, for example, tells you that 25% of the variation in y is accounted for by its linear relationship with x, while 75% remains unexplained.
The standard test asks whether the true slope is zero:
Null hypothesis: H0: β1 = 0 (x has no linear relationship with y).
Alternative hypothesis: Ha: β1 ≠ 0 (x does have a linear relationship with y).
The test statistic is:
T = β̂1 / SE(β̂1)
where:
SE(β̂1) = σ̂ / √[Σ(xi − x̄)²]
and σ̂ is the estimated standard deviation of the residuals (also called the residual standard error).
This T follows a t-distribution with n − 2 degrees of freedom. A small p-value (typically < 0.05) leads you to reject H0 and conclude that the linear relationship is statistically significant.
At a specific value x0, the estimated mean response is β̂0 + β̂1·x0. The (1 − α)·100% confidence interval is:
(β̂0 + β̂1·x0) ± t(n−2, α/2) · SE(β̂0 + β̂1·x0)
where:
SE(β̂0 + β̂1·x0) = σ̂ · √[1/n + (x0 − x̄)² / Σ(xi − x̄)²]
Notice that the interval is narrowest when x0 = x̄ (at the centre of the data) and grows wider as x0 moves away from x̄. This makes intuitive sense: you are most certain about the line near the middle of your data.
To predict an individual y (not just the mean) at x0, you need a prediction interval:
(β̂0 + β̂1·x0) ± t(n−2, α/2) · SE_pred(x0)
where:
SE_pred(x0) = σ̂ · √[1 + 1/n + (x0 − x̄)² / Σ(xi − x̄)²]
The extra "1" under the square root accounts for the inherent variability of individual observations. This is why prediction intervals are always wider than confidence intervals at the same x0.
The population correlation coefficient is:
ρ = Cov(X, Y) / (σX · σY)
The sample estimate (Pearson's r) is:
r = Σ(xi − x̄)(yi − ȳ) / √[Σ(xi − x̄)² · Σ(yi − ȳ)²]
Properties of r:
Ranges from −1 to +1.
r = +1 or −1 indicates a perfect linear relationship.
r = 0 indicates no linear association (there may still be a non-linear one).
r is unitless and symmetric: cor(x, y) = cor(y, x).
In simple linear regression, r² = R².
To test H0: ρ = 0 (no linear relationship) against Ha: ρ ≠ 0, use:
T = r · √(n − 2) / √(1 − r²)
This follows a t-distribution with n − 2 degrees of freedom. In R, the function cor.test(x, y) returns r, a confidence interval for ρ, and the p-value in one call.
Quantity | Formula |
|---|---|
Residual | ei = yi − ŷi |
R² | SSR / SST = 1 − SSE / SST |
SST | Σ(yi − ȳ)² |
SSR | Σ(ŷi − ȳ)² |
SSE | Σ(yi − ŷi)² |
SE of slope | σ̂ / √[Σ(xi − x̄)²] |
t-statistic (slope) | β̂1 / SE(β̂1), df = n − 2 |
CI for mean response | (β̂0 + β̂1·x0) ± t · σ̂ · √[1/n + (x0 − x̄)²/Σ(xi − x̄)²] |
PI for future observation | (β̂0 + β̂1·x0) ± t · σ̂ · √[1 + 1/n + (x0 − x̄)²/Σ(xi − x̄)²] |
Pearson r | Σ(xi − x̄)(yi − ȳ) / √[Σ(xi − x̄)² · Σ(yi − ȳ)²] |
t-statistic (correlation) | r · √(n − 2) / √(1 − r²), df = n − 2 |
Prediction intervals are what you use when individual forecasts matter. A hospital predicting a single patient's recovery time needs a prediction interval, not just a confidence interval for the average patient. The distinction between "average behaviour" and "individual behaviour" shows up anywhere models are used for operational decisions: insurance pricing, weather forecasts, manufacturing tolerances.
Correlation is routinely reported in psychology, economics, and epidemiology as a quick summary of linear association. It is one of the first numbers a data analyst checks when exploring a new dataset.
Students often confuse confidence intervals and prediction intervals. The confidence interval estimates where the line is (the mean response). The prediction interval estimates where a single new observation will fall. The prediction interval is always wider.
A high R² does not mean the model is correct. A model can explain a large share of variation and still violate assumptions or miss a non-linear pattern. Always check residual plots.
Students sometimes interpret r = 0 as "no relationship." It means no linear relationship. A perfect parabolic relationship, for example, can produce r = 0.
Correlation does not imply causation. A strong r between two variables may be driven by a confounding third variable or by coincidence.
⚠️ Expect to be asked to compute R² from SST, SSR, and SSE, or to interpret a given R² value in context.
⚠️ Distinguishing between confidence intervals and prediction intervals (in words and in the formulas) is a very common exam question. Know which has the extra "1" and why.
⚠️ Residual plot interpretation is frequently tested: given a plot, identify whether linearity, normality, or constant variance is violated.
⚠️ Hypothesis testing on the slope: be able to state H0, Ha, compute the t-statistic, and draw a conclusion from a given p-value.
⚠️ Know that in simple linear regression, r² = R². This crops up as a quick-answer or multiple-choice item.
True or false: Prediction intervals are narrower than confidence intervals at the same x0.
Fill in the blank: R² = 0.60 means that ______% of the variation in y is explained by the model.
True or false: If the p-value for the slope is 0.03, you reject H0 at the 5% significance level.
Fill in the blank: The test statistic for the slope follows a t-distribution with ______ degrees of freedom.
True or false: A Pearson correlation of r = 0 guarantees that X and Y are unrelated.
Answers: 1. False (prediction intervals are always wider). 2. 60%. 3. True. 4. n − 2. 5. False (it means no linear relationship; a non-linear relationship may still exist).
Q: A simple linear regression yields SST = 400, SSE = 120. Calculate R² and interpret it.
A: R² = 1 − SSE/SST = 1 − 120/400 = 0.70. About 70% of the variation in y is explained by its linear relationship with x.
Q: Explain in your own words why a prediction interval is wider than a confidence interval.
A: A confidence interval estimates where the average y sits at a given x, so it only reflects uncertainty about the position of the line. A prediction interval estimates where a single future observation will fall, so it must also account for the natural scatter of individual data points around that average. The extra source of variability makes it wider.
Q: A researcher finds r = −0.85 between hours of sleep and number of errors on a task. Interpret this value.
A: There is a strong negative linear association between hours of sleep and errors. As hours of sleep increase, the number of errors tends to decrease. This does not by itself establish that more sleep causes fewer errors.
Q: You are given a residual-vs-fitted-values plot that shows a clear "U" shape. What does this suggest, and what would you do?
A: The "U" shape suggests the linearity assumption is violated: the true relationship between x and y is likely curved. You might consider adding a quadratic term (x²) to the model or applying a transformation to x or y.
Q: State the null and alternative hypotheses for testing whether the population correlation ρ equals zero, and give the test statistic formula.
A: H0: ρ = 0; Ha: ρ ≠ 0. The test statistic is T = r · √(n − 2) / √(1 − r²), which follows a t-distribution with n − 2 degrees of freedom.
Residual analysis techniques carry over directly to multiple regression and ANOVA, where you still rely on the same diagnostic plots. R² extends to adjusted R² when you add more predictors, which penalises unnecessary complexity. The confidence-interval and prediction-interval distinction reappears in time-series forecasting and machine learning prediction intervals. Correlation analysis is a stepping stone to partial correlation and regression with multiple predictors.
residual, residual plot, residual analysis, Q-Q plot, normal probability plot, homoscedasticity, heteroscedasticity, funnel shape, R-squared, R², coefficient of determination, SST, SSR, SSE, total sum of squares, regression sum of squares, error sum of squares, explained variation, unexplained variation, hypothesis test, t-test for slope, standard error of the slope, p-value, confidence interval, prediction interval, mean response, future observation, Pearson correlation, r, rho, ρ, sample correlation, population correlation, covariance, cor.test, strength of association, linear relationship, STAT 302, principles of statistics