Introduction to Probability and Statistics, STAT 3021 | Source: Mathematical Expectation and Variance guide
Difficulty: Intermediate | Prerequisites: Expected value and variance notes (companion document), basic integration
Once you can compute E(X) and Var(X) for a single variable, the next step is handling functions of random variables and relationships between pairs of variables. LOTUS (the Law of the Unconscious Statistician) lets you find E[g(X)] without first deriving the distribution of g(X). Covariance and correlation measure how two random variables move together, which matters as soon as you work with more than one variable at a time. These tools are essential for portfolio theory, regression, and any multivariate analysis later in the course.
LOTUS says you can find the expected value of any function of X by summing (or integrating) that function times the original PMF or PDF, with no need to derive a new distribution. Covariance captures whether two variables tend to increase together or move in opposite directions. The correlation coefficient standardises covariance to a –1 to +1 scale so you can compare the strength of linear relationships across different variable pairs.
Law of the Unconscious Statistician (LOTUS)
A rule for computing E[g(X)] directly from the distribution of X, without first finding the distribution of g(X). For discrete X: E[g(X)] = Σ g(x) · P(X = x). For continuous X: E[g(X)] = ∫ g(x) · f(x) dx.
In simple terms, you can plug your function into the existing PMF or PDF and take the weighted average. No new distribution needed.
Covariance, Cov(X, Y)
A measure of the joint variability of two random variables. Defined as Cov(X, Y) = E[(X – μ_X)(Y – μ_Y)], or equivalently Cov(X, Y) = E(XY) – E(X)E(Y).
Think of it as a number telling you whether X and Y tend to move in the same direction (positive), opposite directions (negative), or show no linear pattern (zero).
Correlation coefficient, ρ_XY
A standardised version of covariance: ρ_XY = Cov(X, Y) / (σ_X · σ_Y). It always lies between –1 and +1.
In simple terms, correlation is covariance stripped of its units and scaled, so you can compare the strength of linear association between any two pairs of variables on the same scale.
Independent random variables
X and Y are independent if knowing the value of one tells you nothing about the other. Independence implies Cov(X, Y) = 0, but the reverse is not always true.
Think of it as: zero covariance means no linear relationship, but there could still be a non-linear one.
The core idea
Suppose you have a random variable X with a known distribution, and you want E[g(X)] for some function g. LOTUS says you do not need to derive the distribution of the new variable g(X). Instead:
Discrete: E[g(X)] = Σ g(x) · P(X = x)
Continuous: E[g(X)] = ∫ g(x) · f(x) dx
This is powerful because deriving the distribution of g(X) can be tedious, whereas LOTUS lets you work directly with the original PMF or PDF.
Why it is called "unconscious"
The name is tongue-in-cheek. The formula looks so natural that students often use it without realising they are invoking a theorem. The point is that the theorem justifies what feels like the obvious calculation.
Special case: quadratic functions
If Y = aX² + bX + c, then E(Y) = a · E(X²) + b · E(X) + c. This follows from linearity of expectation applied to LOTUS. It comes up when computing variance via the shortcut formula, because Var(X) = E(X²) – [E(X)]² requires E(X²).
Worked example (continuous)
A dealer's profit X on a new car has density f(x) = 2(1 – x) for 0 < x < 1 and 0 elsewhere.
Average profit: E(X) = ∫ from 0 to 1 of x · 2(1 – x) dx = ∫ (2x – 2x²) dx = [x² – (2/3)x³] from 0 to 1 = 1/3.
The dealer can expect a profit of about 0.33 units on average.
Definition
Cov(X, Y) = E[(X – μ_X)(Y – μ_Y)]
Computational formula
Cov(X, Y) = E(XY) – E(X) · E(Y)
This is usually easier to work with. Compute E(XY), E(X), and E(Y) separately, then subtract.
Interpretation
Positive covariance: larger X values tend to pair with larger Y values.
Negative covariance: larger X values tend to pair with smaller Y values.
Zero covariance: no linear association between X and Y. This does not rule out a non-linear relationship.
Properties
Cov(X, X) = Var(X)
Cov(X, Y) = Cov(Y, X)
Cov(aX + b, cY + d) = ac · Cov(X, Y). Constants added to the variables do not affect covariance; multiplicative constants scale it.
If X and Y are independent, then Cov(X, Y) = 0. The converse does not hold in general.
Worked example
X and Y have the following joint distribution:
x = –1 | x = 0 | x = 1 | |
|---|---|---|---|
y = 0 | 1/4 | 0 | 1/4 |
y = 1 | 0 | 1/2 | 0 |
E(X) = (–1)(1/4) + (0)(1/2) + (1)(1/4) = 0
E(Y) = (0)(1/2) + (1)(1/2) = 0.5
E(XY) = (–1)(0)(1/4) + (0)(1)(1/2) + (1)(0)(1/4) = 0
Cov(X, Y) = 0 – (0)(0.5) = 0
Covariance is zero here, yet X and Y are not independent (knowing X = 0 forces Y = 1). This is a classic example of zero covariance without independence.
Correlation coefficient
ρ_XY = Cov(X, Y) / (σ_X · σ_Y)
ρ = +1: perfect positive linear relationship. Every point falls on a line with positive slope.
ρ = –1: perfect negative linear relationship.
ρ = 0: no linear relationship (but possibly a non-linear one).
The closer |ρ| is to 1, the tighter the linear pattern.
Variance of a linear combination (independent case)
If X and Y are independent:
Var(aX + bY) = a² Var(X) + b² Var(Y)
General case (not necessarily independent)
Var(aX + bY) = a² Var(X) + b² Var(Y) + 2ab Cov(X, Y)
The covariance term is the piece that drops out under independence. When X and Y have positive covariance, the variance of their sum is larger than the independent case. When they have negative covariance, it is smaller, which is the mathematical basis for diversification in finance.
Formula | Expression |
|---|---|
LOTUS (discrete) | E[g(X)] = Σ g(x) · P(X = x) |
LOTUS (continuous) | E[g(X)] = ∫ g(x) · f(x) dx |
Covariance (definition) | Cov(X, Y) = E[(X – μ_X)(Y – μ_Y)] |
Covariance (shortcut) | Cov(X, Y) = E(XY) – E(X)E(Y) |
Correlation | ρ_XY = Cov(X, Y) / (σ_X · σ_Y) |
Var(aX + bY), independent | a² Var(X) + b² Var(Y) |
Var(aX + bY), general | a² Var(X) + b² Var(Y) + 2ab Cov(X, Y) |
Covariance and correlation are the mathematical backbone of portfolio diversification. By selecting assets whose returns have low or negative covariance, investors reduce the overall variance of the portfolio without necessarily reducing expected return. This is the core insight of modern portfolio theory.
LOTUS is used in actuarial science and engineering whenever you need the average of a transformed variable. For example, if the profit on a sale is a non-linear function of a random input cost, LOTUS lets you compute expected profit directly from the input distribution.
Students often assume that Cov(X, Y) = 0 means X and Y are independent. It does not. Zero covariance rules out a linear relationship only. A quadratic or other non-linear dependence can exist with zero covariance.
Confusing covariance and correlation is common. Covariance depends on the units of X and Y, so its magnitude is hard to interpret alone. Correlation is the dimensionless version, always between –1 and +1.
When applying the variance-of-a-sum formula, students frequently forget the 2ab Cov(X, Y) term in the non-independent case, or accidentally include it when the problem states independence.
With LOTUS, a common error is trying to derive the entire distribution of g(X) first. The whole point of LOTUS is that you do not need to do that. If you find yourself deriving a new PMF or PDF before computing the expected value, you are likely taking the long way round.
⚠️ LOTUS questions often appear as: "Find E[g(X)] given the distribution of X." Practise setting up the sum or integral quickly.
⚠️ The computational formula for covariance, Cov(X, Y) = E(XY) – E(X)E(Y), is usually faster on an exam than the definition formula. Know both, but default to the shortcut.
⚠️ Expect a question asking you to find the variance of a linear combination. The key decision is whether the covariance term is needed. Read the problem for the word "independent" or its absence.
⚠️ A favourite exam trap: giving a joint distribution where Cov(X, Y) = 0, then asking whether X and Y are independent. The answer is not always yes. Check the joint versus marginal factorisation.
True or False: To use LOTUS, you must first find the distribution of g(X).
False. That is exactly what LOTUS lets you skip.
Fill in the blank: Cov(X, Y) = E(XY) – ___.
E(X) · E(Y)
True or False: If ρ_XY = 0, X and Y are independent.
False. Zero correlation means no linear relationship, but non-linear dependence may still exist.
Fill in the blank: Var(X + Y) = Var(X) + Var(Y) + ___.
2 Cov(X, Y)
True or False: Correlation can exceed 1 in absolute value.
False. ρ is always between –1 and +1.
Q: X has density f(x) = 3x² for 0 < x < 1. Find E(X²) using LOTUS.
A: E(X²) = ∫ from 0 to 1 of x² · 3x² dx = ∫ 3x⁴ dx = 3[x⁵/5] from 0 to 1 = 3/5
Q: Given E(X) = 2, E(Y) = 3, and E(XY) = 7, find Cov(X, Y).
A: Cov(X, Y) = E(XY) – E(X)E(Y) = 7 – (2)(3) = 1
Q: If Var(X) = 4, Var(Y) = 9, and Cov(X, Y) = –2, what is Var(X + Y)?
A: Var(X + Y) = 4 + 9 + 2(–2) = 9
Q: Two stocks have σ_X = 5, σ_Y = 8, and Cov(X, Y) = –10. Find the correlation coefficient.
A: ρ = –10 / (5 × 8) = –10/40 = –0.25
Q: X and Y have zero covariance. Does this guarantee independence?
A: No. Zero covariance means no linear relationship. X and Y could still be dependent through a non-linear relationship. The classic example is X symmetric about zero and Y = X².
Covariance is the foundation for the Pearson correlation coefficient used in regression. When you fit a line Y = a + bX, the slope b is Cov(X, Y) / Var(X). Understanding how covariance works here makes the regression formula far less mysterious.
LOTUS connects to moment-generating functions (MGFs), which you may encounter later. An MGF is E[e^(tX)], computed via LOTUS. The variance of linear combinations connects directly to the analysis of estimators: the variance of a sample mean, for example, is derived from the variance-of-a-sum formula applied to independent observations.
LOTUS, Law of the Unconscious Statistician, expected value of a function, E[g(X)], covariance, Cov(X Y), joint variability, correlation coefficient, rho, Pearson correlation, linear combination variance, independence vs zero covariance, portfolio diversification, variance of a sum, E(XY), computational formula for covariance, standardised covariance