Difficulty: Intermediate | Prerequisites: Basic linear algebra, introductory statistics, familiarity with optimization concepts from earlier modules.
This material sits at the intersection of statistics, machine learning and mathematical optimisation. The central idea: the models you already know (regression, classification, clustering, time series) can each be reframed as an optimisation problem with variables, constraints and an objective function. Understanding that reframing is what lets you customise models beyond their textbook defaults and reason about why some problems are computationally harder than others. You should already be comfortable with basic regression and the idea of minimising an error term.
Every major statistical and ML model can be expressed as an optimisation problem. The type of objective function and constraints determines how hard the problem is to solve. Recognising where a model sits on the easy-to-hard spectrum (linear program through to non-convex program) tells you what tools and trade-offs to expect.
Optimisation model
A mathematical formulation with three parts: decision variables (what you solve for), constraints (limits on the variables) and an objective function (what you minimise or maximise). In simple terms, it is the formal way of asking "what is the best choice, given these rules?"
Objective function
The expression you are trying to minimise or maximise. For regression, this is typically the sum of squared errors. Think of it as the score your model is trying to improve.
Constraint
A restriction the solution must satisfy. For example, Lasso regression constrains the sum of absolute coefficient values to be at most T. In simple terms, constraints are the guardrails that keep your solution realistic or well-behaved.
Decision variables
The unknowns the solver adjusts. In a regression optimisation model, the coefficients a_0 through a_m are the decision variables, even though statistics calls them "coefficients" and calls the data columns "variables."
Lasso regression (L1 regularisation)
Ordinary least squares with a constraint on the sum of the absolute values of the coefficients: the sum of |a_j| must be at most T. Think of it as regression that can shrink unimportant coefficients all the way to zero, performing variable selection.
Ridge regression (L2 regularisation)
Ordinary least squares with a constraint on the sum of squared coefficients: the sum of a_j squared must be at most T. In simple terms, it shrinks coefficients towards zero but rarely eliminates them entirely.
Elastic Net regression
A blend of Lasso and Ridge, controlled by a mixing parameter lambda. The constraint is lambda times the L1 term plus (1 minus lambda) times the L2 term, all at most T. Think of it as a compromise that gets some variable selection from Lasso and some stability from Ridge.
Logistic regression
A classification model whose optimisation form maximises the likelihood of observing the actual labels. The predicted probability uses the sigmoid function: p(x_i) = 1 / (1 + e^(minus the linear combination)). In simple terms, it finds the line (or hyperplane) that best separates two classes by maximising how probable the observed outcomes are.
Support vector machine (SVM)
A classification model that finds the widest possible margin between classes. Hard classification enforces that every point is on the correct side; soft classification allows some misclassification by adding a penalty term. Think of it as drawing the fattest possible boundary line between groups.
Hinge loss
The max{0, 1 minus (predicted value times true label)} term in the soft-margin SVM objective. It penalises points that are on the wrong side of the margin or too close to it.
Exponential smoothing
A time-series forecasting method with smoothing parameters alpha, beta and gamma, each constrained between 0 and 1. The objective minimises the sum of squared forecast errors. Think of it as a weighted average that gives more recent observations more influence.
ARIMA (Autoregressive Integrated Moving Average)
A time-series model whose optimisation form finds the parameters mu, phi_i and theta_i that minimise the sum of squared forecast errors, using differenced data D_(d)t. In simple terms, it models a time series as a combination of its own past values and past forecast errors.
GARCH (Generalised Autoregressive Conditional Heteroskedasticity)
A model for forecasting variance (volatility) over time. The optimisation finds omega, beta_i and gamma_i to minimise the squared difference between actual and estimated variance. Think of it as ARIMA, but for how much a series bounces around rather than for its level.
K-means clustering
An unsupervised model that assigns each data point to one of k clusters by minimising total distance to cluster centres. Variables include the cluster centre coordinates z_jk and binary assignment indicators y_ik.
Heuristic
A practical, rule-of-thumb method that finds a feasible (though not necessarily optimal) solution when exact optimisation is too slow or complex. Think of it as a good-enough shortcut.
The standard linear regression objective is to find coefficients a_0, a_1, ..., a_m that minimise the sum of squared residuals. There are no constraints.
Variables: a_0, a_1, ..., a_m
Constraints: none
Objective: minimise the sum from i = 1 to n of (y_i minus (a_0 + sum of a_j * x_ij))^2
A critical terminology trap sits here. In statistics, the x_ij are called "variables" and the a_j are called "coefficients." In the optimisation model, the roles reverse: the x_ij are known constants (your data), and the a_j are the decision variables the solver adjusts.
All three regularised regression models share the same objective function as ordinary linear regression. They differ only in the constraint placed on the coefficients.
Lasso (L1): adds the constraint that the sum of |a_j| is at most T. This tends to force some coefficients to exactly zero, effectively selecting a subset of variables.
Ridge (L2): adds the constraint that the sum of a_j^2 is at most T. This shrinks coefficients towards zero but generally keeps all of them non-zero.
Elastic Net: adds a blended constraint: lambda * sum(|a_j|) + (1 minus lambda) * sum(a_j^2) is at most T. The parameter lambda controls the mix between Lasso-style sparsity and Ridge-style stability.
The threshold T controls how tightly the coefficients are constrained. A smaller T forces more shrinkage. A larger T relaxes the constraint, and in the limit you recover ordinary least squares.
Logistic regression is a classification model. Instead of minimising squared errors, it maximises the likelihood of the observed class labels.
Variables: a_0, a_1, ..., a_m
Constraints: none
Objective: maximise the product of p(x_i) for all i where y_i = 1, times the product of (1 minus p(x_i)) for all i where y_i = 0
Where: p(x_i) = 1 / (1 + e^(minus(a_0 + sum of a_j * x_ij)))
The sigmoid function maps the linear combination to a probability between 0 and 1. The optimisation finds the coefficients that make the observed data most probable under the model.
SVMs come in two flavours, each with a different optimisation formulation.
Every data point must be on the correct side of the decision boundary, with a margin of at least 1.
Variables: a_0, a_1, ..., a_m
Constraints: (a_0 + sum of a_j * x_ij) * y_i >= 1 for each data point i
Objective: maximise the sum of a_j^2 (which is equivalent to maximising the margin width)
Allows some misclassification by introducing a hinge loss penalty instead of hard constraints.
Variables: a_0, a_1, ..., a_m
Constraints: none (the penalty is folded into the objective)
Objective: minimise the sum of max{0, 1 minus (a_0 + sum of a_j * x_ij) * y_i} + lambda * sum of a_j^2
The lambda parameter controls the trade-off between allowing misclassifications (the hinge loss term) and keeping a wide margin (the regularisation term). A larger lambda penalises complex boundaries more heavily.
Exponential smoothing finds the smoothing parameters that produce the best forecasts.
Variables: alpha, beta, gamma
Constraints: each parameter must be between 0 and 1 (inclusive)
Objective: minimise the sum of (x_t minus x-hat_t)^2
The constraints here matter: they keep the smoothing weights sensible. Alpha controls the level, beta the trend and gamma the seasonality.
ARIMA models work with differenced data and combine autoregressive and moving-average terms.
Variables: mu, phi_i, theta_i
Constraints: none
Objective: minimise the sum of (x_t minus x-hat_t)^2
Where: D_(d)t = mu + sum of alpha_i * D_(d)(t-i) minus sum of theta_i * (x-hat_(t-i) minus x_(t-i))
The differencing order d is chosen before optimisation. The solver then finds the autoregressive (phi) and moving-average (theta) parameters.
GARCH models forecast variance rather than level. This is essential for financial data where volatility clusters.
Variables: omega, beta_i, gamma_i
Constraints: none
Objective: minimise the sum of (sigma_t^2 minus sigma-hat_t^2)^2
Where: sigma-hat_t^2 = omega + sum of beta_i * sigma_(t-i)^2 minus sum of gamma_i * epsilon_(t-i)^2
The structure mirrors ARIMA but operates on the variance series instead of the level series.
K-means assigns each of n data points to one of k clusters while finding the best cluster centres.
Variables:
z_jk: coordinate j of cluster centre k
y_ik: 1 if point i is assigned to cluster k, 0 otherwise
Constraints: the sum of y_ik over all clusters k equals 1 for every data point i (each point belongs to exactly one cluster)
Objective: minimise the sum over all clusters k and points i of y_ik times the p-th root of the sum of (x_ij minus z_jk)^p
In practice, p = 2 gives Euclidean distance, which is the standard formulation. The solver alternates between reassigning points to the nearest centre and recomputing centres.
Once you see a model as an optimisation problem, you can customise it in ways the textbook version does not offer.
Change the objective function: alter the exponent on the error term, switch between minimising and maximising, or weight certain data points more heavily.
Add custom constraints: force the y-intercept to a specific value, require coefficients to be non-negative, or require coefficients to be monotonically decreasing.
Use optimisation software: solvers like Gurobi, CPLEX or open-source alternatives (PuLP, SciPy) can handle these modified formulations directly.
This flexibility is the practical payoff of the optimisation perspective. You are not locked into the standard model; you can encode domain knowledge as constraints.
Model | Variables | Constraint | Objective |
|---|---|---|---|
Linear Regression | a_0 ... a_m | None | min sum (y_i minus y-hat_i)^2 |
Lasso | a_0 ... a_m | sum | a_j |
Ridge | a_0 ... a_m | sum a_j^2 <= T | min sum (y_i minus y-hat_i)^2 |
Elastic Net | a_0 ... a_m | lambda * sum | a_j |
Logistic Regression | a_0 ... a_m | None | max likelihood |
SVM (Hard) | a_0 ... a_m | (a_0 + sum a_j * x_ij) * y_i >= 1 | max sum a_j^2 |
SVM (Soft) | a_0 ... a_m | None | min hinge loss + lambda * sum a_j^2 |
Exponential Smoothing | alpha, beta, gamma | 0 <= each <= 1 | min sum (x_t minus x-hat_t)^2 |
ARIMA | mu, phi_i, theta_i | None | min sum (x_t minus x-hat_t)^2 |
GARCH | omega, beta_i, gamma_i | None | min sum (sigma_t^2 minus sigma-hat_t^2)^2 |
K-Means | z_jk, y_ik | sum_k y_ik = 1 for all i | min total distance to centres |
Students often confuse statistical terminology with optimisation terminology. In statistics, x values are "variables" and a values are "coefficients." In the optimisation model it is the other way round: a values are the variables being solved for, and x values are known constants. Getting this backwards will cause confusion in every model formulation.
Students sometimes think Lasso and Ridge do the same thing. They do not. Lasso (L1) can zero out coefficients entirely, performing variable selection. Ridge (L2) shrinks coefficients but keeps them all non-zero. This distinction matters when you have many irrelevant features.
Students often assume the SVM objective is to minimise error. For hard-margin SVM, the objective is to maximise the sum of squared coefficients (which corresponds to maximising the margin width). The classification correctness is enforced through the constraints, not the objective.
Students frequently overlook that exponential smoothing has constraints (0 to 1 on each parameter) while ARIMA and GARCH have none. This means exponential smoothing is a constrained optimisation problem, which affects how it is solved.
⚠️ Be able to write out the variables, constraints and objective function for each model from memory. Exam questions often ask you to "formulate the optimisation model for Ridge regression" or similar.
⚠️ The terminology flip between statistics and optimisation (x vs a as "variables") is a favourite exam question because it tests whether you understand the optimisation perspective.
⚠️ Know the difference between Lasso, Ridge and Elastic Net constraints. Expect a question asking which method performs variable selection and why.
⚠️ Understand the two SVM formulations (hard vs soft) and what the hinge loss term does in the soft version.
⚠️ For time series models, be ready to explain what each parameter controls (alpha for level, beta for trend, gamma for seasonality in exponential smoothing).
⚠️ The k-means constraint (each point assigned to exactly one cluster) is a common exam detail.
True or false: In the optimisation model for linear regression, the x_ij values are the decision variables. (False: a_j values are the decision variables.)
Fill in the blank: Lasso regression adds a constraint on the sum of the ______ of the coefficients. (absolute values)
True or false: The soft-margin SVM has no constraints. (True: the penalty is folded into the objective function via hinge loss.)
Fill in the blank: In exponential smoothing, the parameters alpha, beta and gamma are each constrained to lie between ______ and ______. (0 and 1)
True or false: K-means allows a data point to belong to more than one cluster. (False: the constraint requires each point to be assigned to exactly one cluster.)
Q: Write the optimisation model (variables, constraints, objective) for Ridge regression.
A: Variables: a_0, ..., a_m. Constraint: sum of a_j^2 from j = 1 to m is at most T. Objective: minimise sum from i = 1 to n of (y_i minus (a_0 + sum of a_j * x_ij))^2.
Q: What is the key difference between the Lasso and Ridge constraints, and what practical effect does that difference have?
A: Lasso constrains the sum of absolute values of coefficients (L1 norm), while Ridge constrains the sum of squared coefficients (L2 norm). The L1 constraint can force coefficients to exactly zero, performing automatic variable selection. The L2 constraint shrinks coefficients towards zero but rarely eliminates them.
Q: In the hard-classification SVM formulation, what role do the constraints play?
A: The constraints (a_0 + sum a_j * x_ij) * y_i >= 1 for each i ensure that every data point is correctly classified with a margin of at least 1 from the decision boundary. Classification is enforced by the constraints, while the objective maximises the margin width.
Q: Why does exponential smoothing have constraints while ARIMA does not?
A: Exponential smoothing constrains alpha, beta and gamma to the interval [0, 1] because these are smoothing weights that must be valid proportions. ARIMA parameters (mu, phi_i, theta_i) can take any real value; the model does not restrict their range during optimisation.
Q: Explain the Elastic Net constraint and how the parameter lambda controls the balance between L1 and L2.
A: The Elastic Net constraint is lambda * sum(|a_j|) + (1 minus lambda) * sum(a_j^2) <= T. When lambda = 1, the constraint is pure Lasso (L1). When lambda = 0, it is pure Ridge (L2). Values in between blend the two, getting some variable selection while retaining some stability for correlated features.
Q: What are the decision variables in a k-means optimisation model and what does each represent?
A: There are two sets: z_jk is the j-th coordinate of cluster centre k, and y_ik is a binary indicator that equals 1 if data point i is assigned to cluster k and 0 otherwise.
This material connects directly to earlier modules on basic optimisation and linear programming. The classification of models (linear, quadratic, convex, integer, non-convex) covered in the next section of these notes determines which solver you can use and how long you should expect to wait for a solution.
The regularisation concepts here (Lasso, Ridge, Elastic Net) link to feature engineering and model selection topics in machine learning. If you are studying bias-variance trade-off, regularisation is the mechanism that controls it.
The time series models (exponential smoothing, ARIMA, GARCH) connect to forecasting and financial analytics topics. GARCH in particular is central to risk management in quantitative finance.
Lasso regression is widely used in genomics and medical research to select a small number of relevant genes from thousands of candidates. Ridge regression stabilises predictions when input features are highly correlated, which is common in economic data. SVMs are used in text classification, image recognition and bioinformatics. Exponential smoothing and ARIMA drive demand forecasting in supply chain management. GARCH models are standard in financial risk modelling for estimating Value at Risk.
Tags: optimisation models, linear regression, lasso regression, ridge regression, elastic net, L1 regularisation, L2 regularisation, logistic regression, sigmoid function, support vector machine, SVM, hard margin, soft margin, hinge loss, exponential smoothing, ARIMA, GARCH, k-means clustering, variable selection, prescriptive analytics, objective function, constraints, decision variables