Difficulty: Intermediate | Prerequisites: Chapter 4.1 (expected value), Chapter 4.2 (variance, covariance)

This is Part 3 of 3 covering STAT 3021 Chapter 4. It covers the properties of means and variances under linear combinations, including the critical results for independent random variables. Make sure you are comfortable with E(X), Var(X), and Cov(X,Y) from Parts 1 and 2.

TL;DR

Expectation is linear: E(aX + b) = aE(X) + b, and E(X + Y) = E(X) + E(Y) regardless of dependence. Variance is not linear: adding a constant does not change it, but scaling by a multiplies it by a². For two variables, Var(aX + bY + c) = a²Var(X) + b²Var(Y) + 2abCov(X,Y), and the covariance term drops out when X and Y are independent.

Key Terms

Linearity of expectation

The property that E(aX + b) = aE(X) + b, and more generally E[g(X) ± h(X)] = E[g(X)] ± E[h(X)]. Think of it as: expectation passes through addition, subtraction, and constant multiplication without any conditions on independence.

E(aX + b) = aE(X) + b (Theorem 4.5)

Scaling a random variable by a constant scales the mean by the same constant. Adding a constant shifts the mean by that constant. In simple terms, if everyone's exam score is doubled and then 5 points are added, the class average is also doubled and shifted by 5.

E(X + Y) = E(X) + E(Y) (Corollary 4.3/4.4)

The expected value of a sum is the sum of the expected values, always, whether or not X and Y are independent. This is one of the most useful results in all of probability.

E(XY) = E(X)E(Y) for independent variables (Theorem 4.8)

When X and Y are independent, the expected value of their product factors into the product of their expected values. The converse is false: this factoring can happen even when X and Y are dependent.

Var(aX + c) = a² Var(X) (Corollary 4.6/4.7/4.8)

Adding a constant c to a random variable does not change its variance. Multiplying by a constant a multiplies the variance by a² and the standard deviation by |a|.

Var(aX + bY + c) = a²Var(X) + 2abCov(X,Y) + b²Var(Y) (Theorem 4.9)

The general formula for the variance of a linear combination. When X and Y are independent (Cov = 0), the middle term drops out and variances simply add after scaling.

Core Content

Expectation of a Linear Function (Theorem 4.5)

• E(aX + b) = aE(X) + b. Constants pass through expectation cleanly.

• Corollaries: E(b) = b (expectation of a constant is that constant). E(aX) = aE(X).

• Example: f(x) = x²/3 for −1 < x < 2. E(X) = 5/4. Then E(4X+3) = 4(5/4) + 3 = 8.

Expectation of Sums and Differences (Theorems 4.6, 4.7)

• E[g(X) ± h(X)] = E[g(X)] ± E[h(X)]. This extends to any number of terms and to functions of two variables.

• E[ag(X) ± bh(Y) ± c] = aE[g(X)] ± bE[h(Y)] ± c. Full linearity with constants.

• E[X + Y] = E(X) + E(Y), always. No independence needed.

• Caution: E(XY) ≠ E(X)E(Y) in general. And E(X/Y) ≠ E(X)/E(Y) in general.

• Example: Y = 60X² + 39X + 3. Using linearity, E(Y) = 60E(X²) + 39E(X) + 3. With E(X) = 1 and E(X²) = 7/6, this gives E(Y) = 60(7/6) + 39(1) + 3 = 112 kilowatt hours.

Product of Independent Random Variables (Theorem 4.8)

• If X and Y are independent, then E(XY) = E(X)E(Y).

• More generally, if X and Y are independent, then g(X) and h(Y) are also independent for any functions g and h.

• The converse is false. E(XY) = E(X)E(Y) does not guarantee independence. Example 13 in the notes shows X and Y where E(XY) = E(X)E(Y) = 0 but the joint PMF does not factor, so they are dependent.

• This result also means: for independent X and Y, Cov(X,Y) = E(XY) − E(X)E(Y) = 0.

Variance of a Linear Function (Corollaries 4.6, 4.7, 4.8)

• Var(aX + c) = a² Var(X). The constant c vanishes; the multiplier a gets squared.

• SD(aX + c) = |a| · SD(X).

• Var(−X) = Var(X). Flipping the sign does not change the spread.

• Example: Var(X) = 2. Then Var(−3X + 5) = (−3)² · 2 = 18. SD(−3X + 5) = 3√2.

Variance of a Linear Combination (Theorem 4.9)

• General case: Var(aX + bY + c) = a²Var(X) + 2abCov(X,Y) + b²Var(Y).

• Example: Var(X) = 5, Var(Y) = 3, Cov(X,Y) = 1. Then Var(−2X + 4Y − 3) = 4(5) + 2(−2)(4)(1) + 16(3) = 20 − 16 + 48 = 52.

• Independent case: when Cov(X,Y) = 0, this simplifies to Var(aX + bY + c) = a²Var(X) + b²Var(Y).

• Note: Var(X − Y) = Var(X) + Var(Y) − 2Cov(X,Y). For independent variables, Var(X − Y) = Var(X) + Var(Y). Variances add, they never subtract.

• If Cov(X,Y) > 0, then Var(X+Y) > Var(X) + Var(Y). Positive covariance increases the variance of the sum.

• If Cov(X,Y) < 0, then Var(X+Y) < Var(X) + Var(Y). Negative covariance reduces the variance of the sum. This is the mathematical basis of diversification in finance.

• Extends to n variables: Var(a₁X₁ + ... + aₙXₙ) = Σ aᵢ²Var(Xᵢ) when the Xᵢ are mutually independent.

Formulas

E(aX + b) = aE(X) + b

E[g(X) \pm h(X)] = E[g(X)] \pm E[h(X)]

E(X + Y) = E(X) + E(Y) \quad \text{(always)}

E(XY) = E(X)E(Y) \quad \text{(independent only)}

\text{Var}(aX + c) = a^2 \, \text{Var}(X)

\text{Var}(aX + bY + c) = a^2 \text{Var}(X) + 2ab\,\text{Cov}(X,Y) + b^2 \text{Var}(Y)

\text{Var}(aX + bY + c) = a^2 \text{Var}(X) + b^2 \text{Var}(Y) \quad \text{(independent only)}

\text{Var}(a_1 X_1 + \cdots + a_n X_n) = \sum_{i=1}^{n} a_i^2 \, \text{Var}(X_i) \quad \text{(mutually independent)}

Real-World Applications

Linearity of expectation is the reason a portfolio's expected return is the weighted sum of individual expected returns, no matter how correlated the assets. The variance formula for linear combinations is the reason diversification works: combining negatively correlated assets (negative covariance) reduces portfolio variance below the sum of individual variances.

Common Misconceptions

• Students often write Var(X + Y) = Var(X) + Var(Y) without checking independence. This equality requires Cov(X,Y) = 0. For dependent variables, the 2Cov(X,Y) term can be large.

• Students sometimes believe Var(X − Y) = Var(X) − Var(Y). The correct formula has a plus sign: Var(X−Y) = Var(X) + Var(Y) − 2Cov(X,Y). For independent variables, variances still add.

• Students confuse E(XY) = E(X)E(Y) with independence. This product rule is necessary for independence but not sufficient. Always check the joint distribution against the product of marginals.

• Students forget that E(X/Y) ≠ E(X)/E(Y). Expectation does not pass through division or other non-linear operations.

Why It Matters / Exam Flags

⚠️ The linearity of expectation (E(aX+b) = aE(X)+b, E(X+Y) = E(X)+E(Y)) will appear in nearly every problem. Internalise it.

⚠️ Know the full variance formula Var(aX + bY + c) = a²Var(X) + 2abCov(X,Y) + b²Var(Y), and know when the covariance term drops out.

⚠️ Expect a multiple-choice question asking which statements about expectation and variance are true. Common traps: Var(X−Y) = Var(X) − Var(Y) (false), E(XY) = E(X)E(Y) implies independence (false), E(X/Y) = E(X)/E(Y) (false).

⚠️ SD(aX) = |a|·SD(X), not a²·SD(X). The square on a only applies to variance.

Quick Self-Test

1. True or False: E(X + Y) = E(X) + E(Y) only when X and Y are independent. (False. It holds always.)

1. True or False: Var(X + 10) = Var(X) + 10. (False. Var(X + 10) = Var(X). Adding a constant does not change variance.)

1. Fill in the blank: Var(3X) = ______ Var(X). (9.)

1. True or False: If X and Y are independent, then Var(X − Y) = Var(X) − Var(Y). (False. Var(X−Y) = Var(X) + Var(Y) when independent.)

1. Fill in the blank: E(XY) = E(X)E(Y) requires X and Y to be ______. (Independent.)

Practice Q&A

Q: E(X) = 3, E(Y) = 7. What is E(2X − 3Y + 4)?

A: E(2X − 3Y + 4) = 2(3) − 3(7) + 4 = 6 − 21 + 4 = −11.

Q: Var(X) = 10. What is Var(5X − 2)?

A: Var(5X − 2) = 5² Var(X) = 25(10) = 250.

Q: Var(X) = 4, Var(Y) = 9, Cov(X,Y) = −3. Find Var(X + Y).

A: Var(X+Y) = 4 + 9 + 2(−3) = 7.

Q: X and Y are independent with Var(X) = 5 and Var(Y) = 3. Find Var(−2X + 4Y − 3).

A: Since independent, Var(−2X + 4Y − 3) = (−2)²(5) + (4)²(3) = 20 + 48 = 68.

Q: X and Y are independent with E(X) = 2 and E(Y) = 5. Find E(XY).

A: By Theorem 4.8, E(XY) = E(X)E(Y) = 2(5) = 10.

Q: True or false: E(X² + Y²) = E(X²) + E(Y²).

A: True. Linearity of expectation applies to any sum, including sums of non-linear functions. E(X² + Y²) = E(X²) + E(Y²) always.

Connections to Other Topics

These linearity and variance rules are used constantly in inferential statistics: the mean and variance of the sample mean X-bar are derived directly from E(X₁ + ... + Xₙ) and Var(X₁ + ... + Xₙ). They also form the basis of moment-generating functions (Chapter 4.4, if covered). In regression, the variance of the predicted value comes from the same Var(aX + bY + c) machinery.

Related Terms / Search Tags

linearity of expectation, E(aX+b), E(X+Y), E(XY), independent, Theorem 4.5, Theorem 4.6, Theorem 4.7, Theorem 4.8, Theorem 4.9, Corollary 4.5, Var(aX+c), Var(aX+bY+c), variance of sum, variance of difference, standard deviation of linear combination, uncorrelated, STAT 3021, Chapter 4.3