Difficulty: Intermediate | Prerequisites: Basic probability, summation notation, familiarity with sample means and variances.
Simple linear regression is one of the foundational tools in statistics. It gives you a formal way to describe and quantify the relationship between two variables, one predictor and one response, using a straight line. Everything from predicting sales figures to measuring the effect of a drug dosage builds on this framework. If you have covered descriptive statistics and basic probability, you are ready for this material. If terms like "sample mean" or "variance" feel unfamiliar, revisit those first.
Simple linear regression fits a straight line through data to describe how a response variable y changes with a predictor x. The line is estimated using the least squares method, which minimises the total squared distance between observed and predicted values. The model relies on a set of assumptions (linearity, independence, normality, constant variance) that must hold for the resulting inferences to be valid.
Simple linear regression
A statistical method that models the relationship between one independent variable (x) and one dependent variable (y) as a straight line plus random error. In simple terms, it finds the "best fit" line through a scatterplot of two variables.
Independent variable (predictor, explanatory variable, x)
The variable you use to explain or predict the response. Think of it as the input you control or observe first.
Dependent variable (response variable, y)
The variable you are trying to explain or predict. Think of it as the outcome that changes in response to x.
Intercept (β0)
The expected value of y when x equals zero. In simple terms, it is where the regression line crosses the y-axis. It may or may not have a meaningful real-world interpretation, depending on whether x = 0 falls within the range of your data.
Slope (β1)
The average change in y for each one-unit increase in x. In simple terms, it tells you how steep the line is and in which direction: positive means y goes up as x goes up, negative means y goes down as x goes up.
Error term (εi)
The random component that captures everything the straight line misses. Each observation's error is assumed to be independent and normally distributed with mean zero and constant variance σ².
Least squares method (ordinary least squares, OLS)
The estimation technique that finds the line minimising the sum of squared differences between observed y-values and predicted y-values. Think of it as choosing the line that makes the total "miss" as small as possible.
Fitted value (ŷi)
The value of y predicted by the regression line for a given xi. It sits on the line itself, as opposed to the observed data point which may be above or below.
Normal equations
The pair of equations you get by taking the partial derivatives of the sum of squared errors with respect to β0 and β1 and setting them to zero. Solving these gives you the least squares estimates.
Homoscedasticity (constant variance)
The assumption that the spread of errors is the same at every level of x. In simple terms, the scatter around the line does not fan out or narrow as you move along x.
Heteroscedasticity
The opposite of homoscedasticity: the spread of errors changes across levels of x. A funnel shape in a residual plot is the classic signal.
The model for each observation is:
Yi = β0 + β1·xi + εi
β0 and β1 are the unknown population parameters you want to estimate.
εi is the random error for observation i, assumed to follow N(0, σ²).
The model treats x as fixed (set by the experimenter) or, in observational studies, conditions on the observed x-values.
The goal is to choose estimates b0 and b1 that minimise:
f(b0, b1) = Σ (yi − (b0 + b1·xi))²
Setting the partial derivatives to zero and solving gives the formulas:
β̂1 = Σ(xi − x̄)(yi − ȳ) / Σ(xi − x̄)²
β̂0 = ȳ − β̂1·x̄
where x̄ and ȳ are the sample means of x and y.
The numerator of β̂1 is the sample covariance between x and y (without the 1/(n−1) scaling).
The denominator is the sum of squared deviations of x from its mean.
Once you have β̂1, the intercept β̂0 follows directly by forcing the line through the point (x̄, ȳ).
The resulting fitted line is:
ŷ = β̂0 + β̂1·x
Slope (β̂1): For each additional unit of x, y changes on average by β̂1 units. If β̂1 = −5, then every one-unit increase in x is associated with a five-unit decrease in y, on average.
Intercept (β̂0): The predicted value of y when x = 0. This is meaningful only if x = 0 falls within or near the range of your data. If your x-values run from 50 to 200, the intercept is an extrapolation and should be interpreted with caution.
The validity of all subsequent inference (confidence intervals, hypothesis tests) rests on these:
Linearity: The true relationship between x and y is a straight line.
Independence: The errors εi are independent of one another, with no clustering or time-series structure.
Normality: Each εi follows a normal distribution.
Equal variance (homoscedasticity): The variance σ² is the same for all observations, regardless of the value of x.
Fixed x: The predictor values are treated as known constants, though in practice x may be random and the theory still holds under conditioning.
A convenient mnemonic is LINE: Linearity, Independence, Normality, Equal variance.
Quantity | Formula |
|---|---|
Slope estimate | β̂1 = Σ(xi − x̄)(yi − ȳ) / Σ(xi − x̄)² |
Intercept estimate | β̂0 = ȳ − β̂1·x̄ |
Fitted line | ŷ = β̂0 + β̂1·x |
Objective function (OLS) | f = Σ(yi − b0 − b1·xi)² |
Regression is everywhere: a company might regress advertising spend (x) against revenue (y) to estimate the return on each additional pound spent. In medicine, a researcher might regress drug dosage against blood pressure reduction. The slope tells you the practical "rate of exchange" between the two variables, which is often the number decision-makers care about most.
Students often assume the intercept always has a real-world meaning. It does not, unless x = 0 is within or close to the data range. Interpreting it outside that range is extrapolation.
Students sometimes believe that a significant regression line proves causation. Regression quantifies association; causation requires a properly designed experiment or additional causal reasoning.
The least squares line passes through the point (x̄, ȳ). This is a mathematical fact, not a coincidence, and is worth remembering for exam problems that ask you to verify a calculation.
Violating the assumptions does not necessarily make the line itself wrong, but it does make the standard errors, p-values, and confidence intervals unreliable.
⚠️ You will almost certainly be asked to compute β̂1 and β̂0 from summary statistics (sums, means, sum of squares). Practise the arithmetic.
⚠️ Interpreting the slope in context (not just "it's 3.2" but "for each additional year of education, income increases by $3,200 on average") is a standard exam requirement.
⚠️ Listing and checking the LINE assumptions comes up frequently on both short-answer and free-response questions.
⚠️ Be precise about what x = 0 means for the intercept. Examiners reward students who note when the intercept lacks practical meaning.
True or false: The least squares regression line always passes through the point (x̄, ȳ).
Fill in the blank: The slope β̂1 represents the average change in ______ for a one-unit increase in ______.
True or false: If the errors are heteroscedastic, the least squares estimates are biased.
Fill in the blank: The four key assumptions can be remembered by the mnemonic ______.
True or false: An intercept of 150 is always a meaningful prediction.
Answers: 1. True. 2. y; x. 3. False (the estimates are still unbiased, but the standard errors and inference become unreliable). 4. LINE (Linearity, Independence, Normality, Equal variance). 5. False (it depends on whether x = 0 is within the data range).
Q: A researcher fits a simple linear regression and obtains β̂1 = 2.4 and β̂0 = 10. The predictor is hours of study per week, and the response is exam score. Interpret both estimates in context.
A: For each additional hour of study per week, the exam score increases by 2.4 points on average. A student who studies zero hours per week would be predicted to score 10 points, though this may be an extrapolation if no students in the sample studied zero hours.
Q: Name the four assumptions summarised by the LINE mnemonic, and explain what each one means in one sentence.
A: Linearity (the true relationship is a straight line), Independence (errors are not correlated with one another), Normality (errors follow a normal distribution), Equal variance (the spread of errors is the same at every level of x).
Q: Why does the least squares method square the deviations rather than simply summing them?
A: Squaring ensures that positive and negative deviations do not cancel each other out, and it penalises larger deviations more heavily, producing a unique, mathematically tractable solution.
Q: A dataset has x-values ranging from 20 to 80. The fitted intercept is −12. Should you interpret this value literally?
A: No. Since x = 0 is well outside the observed range, the intercept is an extrapolation and does not have a reliable real-world interpretation in this context.
This material connects directly to multiple regression (STAT 302, later chapters), where you extend the model from one predictor to several. The residual diagnostics covered here carry over almost unchanged. It also connects to correlation analysis (covered in the companion notes), because the slope and the Pearson correlation coefficient are algebraically related: β̂1 = r·(sy/sx).
simple linear regression, SLR, OLS, ordinary least squares, least squares estimation, regression line, best fit line, slope, intercept, β0, β1, beta hat, predictor variable, independent variable, explanatory variable, response variable, dependent variable, error term, residual, LINE assumptions, linearity, independence, normality, homoscedasticity, heteroscedasticity, constant variance, fixed x, sample mean, normal equations, regression model, STAT 302, principles of statistics