Difficulty: Intermediate | Prerequisites: Chapter 4.1 (expected value, LOTUS)
This is Part 2 of 3 covering STAT 3021 Chapter 4. It handles variance, standard deviation, covariance, and the correlation coefficient. You should be comfortable with expected value and LOTUS from Part 1 before reading this.
Variance measures how spread out a distribution is around its mean, computed as the expected squared deviation from µ. Covariance captures whether two random variables move together (positive) or in opposite directions (negative), while the correlation coefficient normalises covariance to a unitless scale between −1 and +1. Independent variables always have zero covariance, but zero covariance does not guarantee independence.
Variance / Var(X) / σ²
The expected value of the squared deviation from the mean: Var(X) = E[(X − µ)²]. Think of it as the average squared distance of outcomes from the centre of the distribution. Always non-negative.
Standard deviation / SD(X) / σ
The positive square root of the variance: σ = √Var(X). In simple terms, it is the typical distance between a randomly drawn value and the mean, expressed in the same units as X.
Shortcut formula for variance
An equivalent expression: Var(X) = E(X²) − [E(X)]². Often easier to compute because you only need E(X) and E(X²), without subtracting µ from every value first.
Covariance / Cov(X,Y) / σ_XY
The expected product of the deviations of X and Y from their respective means: Cov(X,Y) = E[(X − µ_X)(Y − µ_Y)]. In simple terms, it tells you whether X and Y tend to move in the same direction (positive covariance) or opposite directions (negative covariance).
Correlation coefficient / ρ_XY
Covariance divided by the product of the standard deviations: ρ_XY = Cov(X,Y) / (σ_X · σ_Y). Unitless, always between −1 and +1. A value of +1 means a perfect positive linear relationship; −1 means a perfect negative linear relationship; 0 means no linear association.
Uncorrelated
Two random variables with Cov(X,Y) = 0. Independent variables are always uncorrelated, but uncorrelated variables are not necessarily independent.
Var(X) = E[(X − µ)²] = Σ (x − µ)² f(x) (discrete) or ∫ (x − µ)² f(x) dx (continuous).
Step 1: Calculate µ = E(X). Step 2: Plug µ into the variance formula.
The variance is always non-negative. It equals zero only if X is a constant (takes one value with probability 1).
Var(X) = E(X²) − µ². This is usually the faster computation route.
Proof idea (continuous case): expand (x − µ)² = x² − 2µx + µ², then integrate term by term. The cross term gives −2µ·E(X) = −2µ², and the constant term gives µ², so the three terms combine to E(X²) − µ².
Discrete: X in {3,4,5} with f(x) = {0.3, 0.4, 0.3}. Mean µ = 4. Using the definition: Var(X) = (3−4)²(0.3) + (4−4)²(0.4) + (5−4)²(0.3) = 0.6. Using the shortcut: E(X²) = 9(0.3)+16(0.4)+25(0.3) = 16.6, so Var(X) = 16.6 − 16 = 0.6.
Discrete with more spread: Y in {1,2,6,8} with g(y) = {0.4, 0.1, 0.3, 0.2}, also µ = 4. Var(Y) = 8.4. Same mean as X, but much larger variance because the values are more spread out.
Continuous: f(x) = 2(1−x) for 0 < x < 1. Recall E(X) = 1/3. Then Var(X) = E(X²) − (1/3)² = 1/6 − 1/9 = 1/18.
Cov(X,Y) = E[(X − µ_X)(Y − µ_Y)] = E(XY) − µ_Xµ_Y (the alternative/shortcut formula, Theorem 4.4).
Two main uses: (1) computing Var(X + Y) = Var(X) + Var(Y) + 2Cov(X,Y), and (2) computing the correlation coefficient.
Cov(X,X) = Var(X). Covariance of a variable with itself is just its variance.
Cov(aX, bY) = ab·Cov(X,Y). Constants scale out multiplicatively.
If X and Y are independent, Cov(X,Y) = 0 (Corollary 4.5). The converse is false: zero covariance does not imply independence.
Coin tossed twice. X = heads on first toss (0 or 1), Y = total heads (0, 1, or 2). E(X) = 1/2, E(Y) = 1, E(XY) = 3/4. Then Cov(X,Y) = 3/4 − (1/2)(1) = 1/4. Positive covariance makes sense: more heads on the first toss tends to raise the total.
ρ_XY = Cov(X,Y) / (σ_X · σ_Y). Normalises covariance to the range [−1, +1].
ρ = +1: perfect positive linear relationship (e.g. Y = aX + b with a > 0).
ρ = −1: perfect negative linear relationship.
ρ = 0: no linear association (but possibly a non-linear one).
The components needed to compute ρ are E(XY), E(X²), E(Y²), E(X), and E(Y).
Calculating covariance and correlation by hand is not required in STAT 3021 (Spring 2024). Focus on definitions, properties, and interpretation.
\sigma^2 = \text{Var}(X) = E[(X - \mu)^2] = \sum_x (x - \mu)^2 f(x) \quad \text{(discrete)}\sigma^2 = \text{Var}(X) = E[(X - \mu)^2] = \int_{-\infty}^{\infty} (x - \mu)^2 f(x)\,dx \quad \text{(continuous)}\sigma^2 = E(X^2) - \mu^2 \quad \text{(shortcut formula)}\sigma = \text{SD}(X) = \sqrt{\text{Var}(X)}\sigma_{XY} = \text{Cov}(X,Y) = E[(X - \mu_X)(Y - \mu_Y)] = E(XY) - \mu_X \mu_Y\rho_{XY} = \frac{\text{Cov}(X,Y)}{\sigma_X \, \sigma_Y} = \frac{\sigma_{XY}}{\sqrt{\text{Var}(X)\,\text{Var}(Y)}}Variance and standard deviation are used everywhere risk is quantified. In finance, portfolio risk is measured by the standard deviation of returns. In manufacturing, quality control uses variance to judge whether a process stays within tolerance. Covariance and correlation underpin regression analysis, which you will encounter in later statistics courses.
Students often think Var(X − Y) = Var(X) − Var(Y). This is wrong. Var(X − Y) = Var(X) + Var(Y) − 2Cov(X,Y). Even when X and Y are independent, the variances add, they do not subtract.
Students sometimes assume Cov(X,Y) = 0 means X and Y are independent. It does not. Zero covariance only rules out linear dependence. Non-linear dependence can exist with zero covariance (see Example 13 in the notes).
Students confuse the units. Variance is in squared units of X (e.g. dollars²). Standard deviation restores the original units. Covariance is in (unit of X)(unit of Y). Correlation is unitless.
The shortcut formula Var(X) = E(X²) − µ² is not the same as E(X²) − E(X). You must square the mean, not just subtract it.
⚠️ Be ready to compute variance using both the definition and the shortcut formula. The shortcut is usually faster.
⚠️ Expect a conceptual question on whether Cov(X,Y) = 0 implies independence. The answer is no.
⚠️ Know the relationship Var(X+Y) = Var(X) + Var(Y) + 2Cov(X,Y), and its special case for independent variables.
⚠️ Interpreting the sign and magnitude of ρ is fair game. Remember: ρ = ±1 means a perfect linear relationship, not just any strong relationship.
True or False: Variance can be negative. (False. It is the expected value of a squared quantity, so it is always ≥ 0.)
Fill in the blank: The shortcut formula for variance is Var(X) = E(X²) − ______. (µ², i.e. [E(X)]².)
True or False: If Cov(X,Y) = 0, then X and Y must be independent. (False.)
True or False: ρ_XY = 2 is a possible correlation value. (False. Correlation is always between −1 and +1.)
Fill in the blank: Cov(X,X) = ______. (Var(X).)
Q: X has distribution: x = {1, 3, 5} with f(x) = {0.25, 0.50, 0.25}. Find Var(X).
A: E(X) = 1(0.25) + 3(0.50) + 5(0.25) = 3. E(X²) = 1(0.25) + 9(0.50) + 25(0.25) = 11. Var(X) = 11 − 9 = 2.
Q: If Var(X) = 4 and Var(Y) = 9, and X and Y are independent, what is Var(X + Y)?
A: Since X and Y are independent, Cov(X,Y) = 0. Var(X+Y) = 4 + 9 = 13.
Q: Same variances as above, but now Cov(X,Y) = 2. What is Var(X − Y)?
A: Var(X−Y) = Var(X) + Var(Y) − 2Cov(X,Y) = 4 + 9 − 4 = 9.
Q: E(XY) = 10, E(X) = 2, E(Y) = 4. What is Cov(X,Y)?
A: Cov(X,Y) = E(XY) − E(X)E(Y) = 10 − (2)(4) = 2.
Q: Which of the following is true? (A) Var(X−Y) = Var(X) − Var(Y). (B) E(XY) = E(X)E(Y) implies independence. (C) E(X/Y) = E(X)/E(Y). (D) None of the above.
A: (D) None of the above. All three statements are false.
Variance feeds directly into Section 4.3 (linear combinations), where you will learn Var(aX + bY + c). Covariance reappears in regression (later courses) as the numerator of the slope coefficient. The shortcut formula Var(X) = E(X²) − µ² is used constantly when deriving the variance of named distributions (binomial, Poisson, normal).
Related Terms / Search Tags
variance, standard deviation, Var(X), sigma, σ², σ, SD, spread, dispersion, covariance, Cov(X,Y), σ_XY, correlation, correlation coefficient, ρ, rho, uncorrelated, independent, shortcut formula, E(X²) minus mu squared, STAT 3021, Chapter 4.2