Source: STAT 212 Lecture Notes, Texas A&M University Difficulty: Intermediate | Prerequisites: Polynomial regression (Parts 1 and 2 of these notes), properties of logarithms
Tags: nonlinear regression, variable transformation, log transformation, multiplicative error, exponential model, power model, linearisation, heteroscedasticity, non-constant variance, STAT 212
Not every relationship is well described by a polynomial. Sometimes the underlying science or the shape of the data suggests a specific functional form, such as an exponential or power model. These are non-polynomial regression models, and their parameters can still be estimated by least squares, though the resulting equations are typically nonlinear and harder to solve. A useful shortcut is variable transformation: by taking logs of Y and/or x, certain non-polynomial models reduce to the familiar straight-line form, allowing you to use all the tools you already know. Transformations also help when the variance of Y changes with x, a violation of the constant-variance assumption that log transformations can often fix.
When the regression function has a known non-polynomial form (exponential, power law, etc.), least squares still applies but the equations become nonlinear. Log transformations can convert multiplicative-error models like Y = αe^{βx}ε and Y = αx^βε into straight-line models in log Y vs. x or log Y vs. log x respectively. Taking logs also stabilises variance when the spread of Y increases with x.
Non-polynomial model
A regression model where the regression function f(x; θ₁, …, θₖ) is not a polynomial. Examples: f = exp(θ₁ + θ₂x + θ₃x²) or f = θ₁cos(2πθ₂x). The functional form is chosen based on subject-matter knowledge.
Nonlinear least squares
The extension of least squares to models where the normal equations are nonlinear in the parameters. The objective is still to minimise Σ[yᵢ − f(xᵢ; a₁, …, aₖ)]², but solving for the parameter estimates requires iterative numerical methods rather than a simple matrix formula.
Multiplicative error
An error structure of the form Y = f(x) × ε, where ε is a positive-valued random variable. This contrasts with the usual additive error Y = f(x) + ε. Multiplicative errors are common in financial and biological data where the variability scales with the level of the response.
Additive error
The standard error structure Y = f(x) + ε, where ε has mean zero and constant variance σ². This is the assumption behind ordinary least squares.
Log transformation
Replacing Y with log Y (and/or x with log x) to convert a nonlinear model into a linear one. In simple terms, you are changing the scale of measurement so that the relationship becomes a straight line.
Exponential model
Y = αe^{βx}ε. Taking logs gives log Y = log α + βx + log ε, which is a straight-line model in log Y vs. x.
Power model (power law)
Y = αx^βε (for x > 0). Taking logs gives log Y = log α + β log x + log ε, which is a straight-line model in log Y vs. log x.
Heteroscedasticity (non-constant variance)
A situation where the variance of Y depends on x. In the power model Y = αx^βε, Var(Y) = α²x^{2β}Var(ε), which increases with x even if Var(ε) is constant. Log transformations often remedy this because Var(log Y) = Var(log ε), which does not depend on x.
Sometimes subject-matter knowledge dictates a specific functional form. For example, radioactive decay follows an exponential model, and allometric relationships in biology follow power laws.
In these cases, forcing a polynomial may require an impractically high degree and still give a poor fit outside the observed range of x.
The general framework is Y = f(x; θ₁, …, θₖ) + ε, where f is a known function up to the parameter values θ₁, …, θₖ.
The objective is still to minimise Σᵢ₌₁ⁿ [yᵢ − f(xᵢ; a₁, …, aₖ)]².
Setting partial derivatives to zero gives a system of equations that are generally nonlinear in the parameters.
These nonlinear equations must be solved numerically (e.g., using iterative algorithms such as Gauss-Newton). This is more computationally demanding than the polynomial case.
Some non-polynomial models can be converted ("linearised") into simple linear regression by transforming Y and/or x.
This avoids the need for nonlinear least squares entirely.
The error here is multiplicative: Y is the product of the mean function αe^{βx} and the random variable ε.
Taking natural logs: log Y = log α + βx + log ε.
Setting Y' = log Y, β₀ = log α, β₁ = β, ε' = log ε gives: Y' = β₀ + β₁x + ε'.
This is a standard straight-line model. Fit it using OLS on the transformed data (log yᵢ vs. xᵢ).
Taking natural logs: log Y = log α + β log x + log ε.
Setting Y' = log Y, x' = log x, β₀ = log α, β₁ = β, ε' = log ε gives: Y' = β₀ + β₁x' + ε'.
Again a straight-line model, now fitted to (log xᵢ, log yᵢ) pairs.
For the power model Y = αx^βε, even if Var(ε) is constant, Var(Y) = α²x^{2β}Var(ε), which depends on x. The variance of Y increases (or decreases) systematically with x.
After the log transformation, Var(log Y) = Var(log ε), which is constant for all x (provided the distribution of ε does not depend on x).
The practical signal: if a residual plot shows the spread of residuals fanning out (or funnelling in) as x increases, a log transformation of Y is worth trying.
The Beveridge wheat price index (1500 to 1869) shows prices increasing over time with variance that also increases.
On the raw scale, a polynomial fit has residuals whose spread grows with year.
On the log scale, the relationship becomes more nearly linear and the residuals have much more uniform spread.
This illustrates the dual benefit of log transforms: linearising the relationship and stabilising variance.
Exponential model and its linearisation: Y = αe^{βx}ε → log Y = log α + βx + log ε
Power model and its linearisation: Y = αx^βε → log Y = log α + β log x + log ε
Variance under the power model: Var(Y) = α²x^{2β}Var(ε) (depends on x) Var(log Y) = Var(log ε) (does not depend on x)
Exponential models appear throughout the sciences: population growth, radioactive decay, compound interest, and the spread of infections all follow (at least approximately) exponential patterns. Power models describe relationships where doubling one quantity multiplies another by a fixed factor, which is common in physics (e.g., gravitational force vs. distance), biology (metabolic rate vs. body mass), and economics (Pareto distributions of income). In each case, working on the log scale makes the analysis both easier and more statistically sound.
Students sometimes confuse multiplicative and additive errors. In an additive-error model, the noise is added to the signal. In a multiplicative-error model, the noise scales the signal. The log transformation converts multiplicative errors into additive ones, which is precisely why it works.
A common mistake is to take logs and then interpret the regression coefficients as if you had never transformed. After fitting log Y = β₀ + β₁x + ε', the coefficient β₁ describes the effect of x on log Y, not on Y directly. To recover the effect on Y, you exponentiate.
Some students believe log transforms are always appropriate when a straight line does not fit. They are not. Log transforms work for specific model structures (exponential, power) with multiplicative errors. For other types of nonlinearity, polynomial regression or another approach may be more suitable.
Students occasionally apply a log transformation to data that contains zeros or negative values. Logarithms are only defined for positive numbers. Check your data before transforming.
⚠️ Be able to linearise both the exponential model (Y = αe^{βx}ε) and the power model (Y = αx^βε) by taking logs. This is a standard exam question.
⚠️ Know the difference between additive and multiplicative error structures and how the log transformation converts one into the other.
⚠️ If an exam question describes residuals with increasing spread, the expected answer is "consider a log transformation of Y."
⚠️ Remember that fitting on the log scale gives coefficients for log Y, not Y. You must exponentiate to get back to the original scale.
⚠️ The wheat price example is a classic illustration of variance stabilisation through log transformation. Be prepared to describe why the log-scale residual plot looks better than the raw-scale one.
True or false: In the model Y = αe^{βx}ε, the error is additive.
A: False. The error is multiplicative.
Fill in the blank: Taking the log of both sides of Y = αx^βε gives log Y = log α + β ____ + log ε.
A: log x.
True or false: Var(log Y) depends on x in the power model Y = αx^βε, provided ε has a fixed distribution.
A: False. Var(log Y) = Var(log ε), which is constant.
Fill in the blank: If residuals from a regression show increasing spread as x grows, a ____ transformation of Y may help.
A: Log (logarithmic).
Q: Starting from the model Y = αe^{βx}ε, derive the linearised form and identify the new intercept, slope, and error term.
A: Take logs of both sides: log Y = log α + βx + log ε. Define Y' = log Y, β₀ = log α, β₁ = β, and ε' = log ε. The linearised model is Y' = β₀ + β₁x + ε', where β₀ is the intercept, β₁ is the slope, and ε' is the error.
Q: A scatterplot of y vs. x shows the data fanning out to the right (greater spread at larger x values). What does this suggest, and what would you do?
A: The increasing spread indicates heteroscedasticity: the variance of Y is growing with x. This is consistent with a multiplicative error structure. A log transformation of Y would stabilise the variance and potentially linearise the relationship.
Q: In the power model Y = αx^βε, why does Var(Y) depend on x even if Var(ε) does not?
A: Var(Y) = Var(αx^βε) = (αx^β)²Var(ε) = α²x^{2β}Var(ε). The factor x^{2β} makes the variance a function of x. The multiplicative structure means Y's variability scales with its expected value.
Q: After fitting log Y = 2.5 + 0.8x on data with natural logs, what is the estimated value of α in the original model Y = αe^{βx}ε?
A: The intercept 2.5 equals log α, so α = e^{2.5} ≈ 12.18. The slope 0.8 equals β directly.
Q: Why does fitting the model on the log scale count as "ordinary least squares" even though the original model is nonlinear?
A: After the log transformation, the model log Y = log α + βx + log ε is linear in the parameters (log α and β). OLS applies to models that are linear in the parameters, regardless of whether the original formulation was nonlinear. The transformation does the work of linearisation.
Log transformations connect to the broader theme of data transformations in statistics, including the Box-Cox family of power transformations covered in more advanced courses.
The multiplicative error structure reappears in generalised linear models (GLMs), where the log link function handles similar situations in a more formal framework.
The variance-stabilisation idea also links to weighted least squares, an alternative to transformation when you want to stay on the original scale but account for non-constant variance.
non-polynomial regression, nonlinear regression, nonlinear least squares, variable transformation, log transformation, natural logarithm, exponential model, power model, power law, multiplicative error, additive error, linearisation, heteroscedasticity, non-constant variance, variance stabilisation, Beveridge wheat price index, fan shape residuals, Gauss-Newton, iterative estimation, STAT 212, Texas A&M