Source: Principles of Statistics I (Texas A&M), Tamhane/Dunlop Ch. 10–11
Tags: Pearson correlation, Spearman rank correlation, correlation coefficient, r, rho, bivariate normal, covariance, association vs causation, extrapolation, interpolation, outlier, studentized residual, leverage, influential point, Cook's distance, spurious correlation
The Pearson correlation coefficient r measures the strength of a linear relationship between X and Y on a unit-free scale from −1 to +1. Spearman's rank correlation is a nonparametric alternative that captures monotonic (not just linear) association. Beyond estimation, regression results can be misused in several ways: confusing association with causation, extrapolating outside the observed data range, ignoring outliers, and being misled by influential data points.
Covariance (Cov(X, Y))
Cov(X, Y) = E[(X − μX)(Y − μY)]. Measures how X and Y move together. Positive when both tend to be above or below their means simultaneously. The value depends on the units of X and Y, making raw covariance hard to interpret on its own.
Correlation coefficient (ρ)
ρ = Cov(X, Y) / (σX · σY). A standardised, unit-free version of covariance. Ranges from −1 to +1. ρ = ±1 means a perfect linear relationship; ρ = 0 means no linear association (though a non-linear one may still exist).
Pearson's product-moment correlation (r)
The sample estimate of ρ, computed as r = Σ(Xᵢ − X̄)(Yᵢ − Ȳ) / (SX · SY). Equivalently, r = β̂₁ · SX / SY, or r = ±√R² with the sign matching β̂₁.
Bivariate normal distribution
A joint distribution of (X, Y) where both marginals are normal and the relationship between them is fully described by ρ. Formal inference on ρ (using the t-test) requires this assumption.
Spearman's rank correlation (rSP)
A nonparametric alternative to Pearson's r. Computed by replacing X and Y values with their ranks and then calculating the usual Pearson r on the ranks. Measures monotonic association, not just linear. Does not require bivariate normality.
Outlier (in regression)
An observation whose residual is unusually large relative to its standard error. Formally, a point where the studentized residual |Tᵢ| > 3.
Studentized residual (Tᵢ)
Tᵢ = eᵢ / SE(eᵢ), where SE(eᵢ) = σ̂ · √(1 − hᵢ). Under correct model conditions, Tᵢ follows a t-distribution with df = n − 2.
Leverage (hᵢ)
hᵢ = (Xᵢ − X̄)² / Σ(Xᵢ − X̄)² + 1/n. Measures how far Xᵢ is from the centre of the X data. High-leverage points (hᵢ > 4/n) can disproportionately influence the fitted line.
Influential point
A data point that substantially changes the estimated regression line when removed. High leverage combined with a large residual produces high influence. Cook's distance is one formal measure.
Interpolation
Estimating μ(Y|X) or ŷ for X values within the observed range of the data. Generally reliable.
Extrapolation
Estimating μ(Y|X) or ŷ for X values outside the observed range. Risky, because the true relationship may be non-linear beyond the data.
Covariance captures the direction of the linear relationship between X and Y, but its magnitude depends on whatever units X and Y happen to use. To get a pure measure of linear association, divide by both standard deviations:
ρ = Cov(X, Y) / (σX · σY)
This gives a quantity bounded between −1 and +1 regardless of scale.
The sample version of ρ is:
r = Σ(Xᵢ − X̄)(Yᵢ − Ȳ) / (SX · SY)
Two useful shortcuts connect r to the regression output:
r = β̂₁ · (SX / SY)
r = +√R² when β̂₁ > 0, and r = −√R² when β̂₁ < 0
In the energy-consumption example (25 months of temperature vs energy use), β̂₁ = −0.0798 and R² = 0.7144, so r = −√0.7144 = −0.845. A strong negative linear association.
Under bivariate normality, the test statistic for H₀: ρ = 0 vs Hₐ: ρ ≠ 0 is:
t = r√(n − 2) / √(1 − r²), with df = n − 2
Statistical vs practical significance: with very large samples, even tiny correlations become statistically significant. The handout gives an example where n = 40,000 and r = 0.02 produces t = 4.0 (p-value = 0.00006), a highly significant result. Yet r = 0.02 represents almost no practical linear relationship. Always look at the size of r alongside the p-value.
When bivariate normality cannot be assumed, or when you want to detect monotonic (not just linear) relationships, use Spearman's rSP.
The procedure:
Replace each Xᵢ with its rank Rᵢ among the X values
Replace each Yᵢ with its rank Sᵢ among the Y values
Compute the Pearson r on the (Rᵢ, Sᵢ) pairs
To test H₀: X and Y are independent, use Z = rSP · √(n − 1), which is approximately standard normal.
In the age-at-first-word vs aptitude-score example (n = 21), Pearson r = −0.640 (p = 0.002) but Spearman rSP = −0.317 (p = 0.162). This discrepancy arose because one extreme data point (case 18, age = 42) inflated the Pearson correlation. Removing it changed r from −0.640 to −0.335. Spearman was more robust to this outlier.
In a randomised, controlled experiment (e.g. randomly assigning drug doses), you can speak of cause and effect.
In an observational study (e.g. comparing wine consumption and heart disease across countries), you cannot. Many confounding variables (diet, lifestyle, environment) may drive both X and Y. The correct term is association, not causation.
Classic example of spurious correlation: the winning percentage of the New York Yankees and annual coal production in England were strongly correlated over the same time period, despite having no causal link.
Interpolation means predicting Y at X values within the observed range. For the dose-strength data (doses 2 to 32), estimating strength at dose 10 or 20 is interpolation and is generally safe.
Extrapolation means predicting Y at X values outside the observed range. Estimating strength at dose 0.5 or dose 50 is extrapolation. The true relationship may be highly non-linear beyond the data, making such predictions unreliable. Extrapolation should be avoided unless you have strong theoretical reasons and specialised models (e.g. time-series forecasting).
An outlier is a data point with an unusually large residual. The formal tool is the studentized residual:
Tᵢ = eᵢ / (σ̂ · √(1 − hᵢ))
where hᵢ = (Xᵢ − X̄)² / Σ(Xᵢ − X̄)² + 1/n.
Under the model assumptions, Tᵢ ~ t(n − 2). A point is flagged as an outlier if |Tᵢ| > 3.
The general rule for handling outliers: do not automatically delete. Remove the point only if there is a documented reason (recording error, unusual experimental conditions, equipment failure). Always note that the point was in the original data and explain why it was excluded.
A high-leverage point has an X value far from the mean of X. Its leverage value hᵢ will be large relative to the average leverage h̄ = 2/n. A common threshold: hᵢ > 4/n signals high leverage.
High leverage alone does not mean the point is harmful, but when combined with a large residual, it can pull the regression line substantially. Removing such a point may change both β̂₀ and β̂₁ markedly.
Cook's distance combines leverage and residual size into a single influence measure. In the age-vs-aptitude example, case 18 (age = 42, far from the cluster of ages 7–26) had extreme leverage and dramatically changed the fitted line when included vs excluded.
With a single explanatory variable, you can often spot influential points visually on the scatterplot. With multiple explanatory variables, formal diagnostics become essential.
Quantity | Formula |
|---|---|
Population correlation | ρ = Cov(X, Y) / (σX · σY) |
Pearson sample r | Σ(Xᵢ − X̄)(Yᵢ − Ȳ) / (SX · SY) |
r from regression | r = ±√R², sign matches β̂₁ |
Test for ρ = 0 | t = r√(n − 2) / √(1 − r²), df = n − 2 |
Spearman rSP | Pearson r computed on ranks |
Test for rSP | Z = rSP · √(n − 1), approximately N(0,1) |
Studentized residual | Tᵢ = eᵢ / (σ̂√(1 − hᵢ)) |
Leverage | hᵢ = (Xᵢ − X̄)² / Σ(Xᵢ − X̄)² + 1/n |
High leverage threshold | hᵢ > 4/n |
Outlier threshold | |Tᵢ| > 3 |
⚠️ r measures linear association only. A strong curved pattern can produce r near zero. Always look at the scatterplot.
⚠️ Statistical significance of r does not imply practical importance. With large n, trivial correlations become "significant."
⚠️ Spearman's rSP is more robust to outliers and does not require bivariate normality. Know when to prefer it over Pearson's r.
⚠️ Correlation does not imply causation. This is one of the most frequently tested conceptual points. Only randomised experiments support causal language.
⚠️ Never extrapolate beyond the observed X range and call it a reliable estimate. The model has no information about what happens outside its data.
⚠️ Do not automatically delete outliers. Only remove them with documented justification, then refit and compare.
⚠️ High leverage alone is not a problem. It becomes a problem when the point also has a large residual, pulling the line away from the bulk of the data.
Q: What is the key difference between Pearson's r and Spearman's rSP?
A: Pearson's r measures the strength of the linear relationship and assumes bivariate normality for inference. Spearman's rSP measures monotonic association (increasing or decreasing, not necessarily linear) and is a rank-based, nonparametric method that does not require normality.
Q: A study of 40,000 workers finds r = 0.02 between a skills test and productivity, with p-value = 0.00006. Should the firm adopt the test as a predictor?
A: Probably not. While the correlation is statistically significant (p < 0.001), r = 0.02 indicates virtually no practical linear relationship. The test explains less than 0.04% of the variation in productivity. With very large samples, even meaningless correlations can achieve statistical significance.
Q: In the age-vs-aptitude data, Pearson r = −0.640 but Spearman rSP = −0.317. Why the large discrepancy?
A: Case 18 (age = 42, score = 57) is an extreme outlier in the X direction. It has very high leverage and inflates the Pearson correlation. Spearman's rank-based method is less sensitive to this single extreme point, giving a more representative measure of the overall monotonic trend.
Q: A researcher fits a SLR model to data with X ranging from 10 to 50, then uses the model to predict Y when X = 100. What is this called, and why is it risky?
A: This is extrapolation. It is risky because the linear relationship observed between X = 10 and X = 50 may not hold outside that range. The true relationship could curve, level off, or reverse direction beyond the observed data.
Q: What conditions must be met before you remove an outlier from a regression analysis?
A: The studentized residual |Tᵢ| should exceed 3, and there must be a substantive reason for removal, such as a recording error, equipment malfunction, or experimental conditions that differed from the rest of the data. You should also document the original data and explain the deletion.
Q: What does a leverage value of hᵢ > 4/n indicate?
A: It indicates that the X value for observation i is far from the mean of X, giving that point disproportionate potential to influence the slope and intercept. Whether it actually distorts the line depends on whether it also has a large residual.
Pearson correlation, product-moment correlation, sample correlation coefficient, Spearman rank correlation, rSP, covariance, bivariate normal, correlation coefficient rho, unit-free measure, linear association, monotonic association, t-test for correlation, statistical significance vs practical significance, association vs causation, confounding variable, spurious correlation, interpolation, extrapolation, outlier in regression, studentized residual, leverage, high leverage point, influential point, Cook's distance, regression diagnostics, robust correlation