Difficulty: Intermediate to Advanced | Prerequisites: Part 1 (single-variable transformations), linear algebra (matrix multiplication, determinants, inverses), multivariable calculus (partial derivatives).
This document picks up where Part 1 left off. When you have more than one random variable, you often need to find the joint distribution of a set of transformed variables. The machinery is the same in spirit (find the c.d.f. or use a change-of-variables formula), but the Jacobian determinant replaces the single derivative. Linear transformations (Y = AX) are the most common special case and show up constantly in multivariate normal theory and regression.
TL;DR
For functions of multiple random variables, the joint p.d.f. of the transformed variables comes from summing (discrete) or applying the multivariate change-of-variables formula (continuous). The formula requires the transformation to be one-to-one with a differentiable inverse, and the key ingredient is the absolute value of the Jacobian determinant. For linear transformations Y = AX, the Jacobian simplifies to 1/|det(A)|.
Joint probability density function (joint p.d.f.)
The function fₓ₁,...,ₓₙ(x₁,...,xₙ) that describes the simultaneous probability distribution of n random variables. In simple terms, it tells you how likely any particular combination of values is.
Jacobian matrix (J)
The matrix of all first-order partial derivatives of the inverse transformation. If the inverse maps (y₁,...,yₙ) to (x₁,...,xₙ) via functions s₁,...,sₙ, then entry (i, j) of J is ∂sᵢ/∂yⱼ. Think of it as a matrix that captures how each input variable responds to changes in each output variable.
Jacobian determinant, |det(J)|
The absolute value of the determinant of the Jacobian matrix. It scales the original density to account for how the transformation stretches or compresses n-dimensional volume. This is the multivariate version of |ds/dy| from Part 1.
Determinant of a 2×2 matrix
For A = [[a, b], [c, d]], det(A) = ad − bc. This is the building block for computing Jacobians in two-variable problems.
Support of a distribution, B
The set of values where the density is non-zero. When transforming variables, the support of Y is the image of the support of X under the transformation.
Linear transformation Y = AX
A transformation where Y is obtained by multiplying a vector of random variables X by a constant matrix A. Requires det(A) ≠ 0 (A is invertible) for the change-of-variables formula to apply.
Suppose X₁,...,Xₙ are discrete random variables with joint p.m.f. fₓ₁,...,ₓₙ(x₁,...,xₙ), and you define m new random variables through m functions:
Y₁ = u₁(X₁,...,Xₙ)
Y₂ = u₂(X₁,...,Xₙ)
...
Yₘ = uₘ(X₁,...,Xₙ)
To find the joint p.m.f. of Y₁,...,Yₘ, define the set Aᵧ as all (x₁,...,xₙ) that map to the particular values (y₁,...,yₘ):
Aᵧ = {(x₁,...,xₙ) : u₁(x₁,...,xₙ) = y₁, ..., uₘ(x₁,...,xₙ) = yₘ}
Then the joint p.m.f. is:
fᵧ₁,...,ᵧₘ(y₁,...,yₘ) = Σ fₓ₁,...,ₓₙ(x₁,...,xₙ), summed over all (x₁,...,xₙ) ∈ Aᵧ.
This is the direct generalisation of the single-variable discrete case: collect all input combinations that produce the desired output, and sum their probabilities.
The transformation from (X₁,...,Xₙ) to (Y₁,...,Yₙ) must be:
One-to-one (each (y₁,...,yₙ) comes from exactly one (x₁,...,xₙ)).
The inverse functions s₁,...,sₙ must be differentiable.
Note that the number of output variables must equal the number of input variables (n = n) for this formula.
The joint p.d.f. of Y₁,...,Yₙ is:
fᵧ₁,...,ᵧₙ(y₁,...,yₙ) = fₓ₁,...,ₓₙ(s₁(y),...,sₙ(y)) · |det(J)| for y ∈ B, and 0 otherwise.
Here B is the support of (Y₁,...,Yₙ) (the image of the support of X under the transformation), and J is the Jacobian matrix with entries Jᵢⱼ = ∂sᵢ/∂yⱼ.
For a two-variable transformation with inverse (x₁, x₂) = (s₁(y₁, y₂), s₂(y₁, y₂)):
J = [[∂s₁/∂y₁, ∂s₁/∂y₂], [∂s₂/∂y₁, ∂s₂/∂y₂]]
det(J) = (∂s₁/∂y₁)(∂s₂/∂y₂) − (∂s₁/∂y₂)(∂s₂/∂y₁)
The absolute value |det(J)| measures how much the inverse transformation stretches or compresses area (or volume, in higher dimensions). This is the direct analogue of |ds/dy| from the single-variable case.
Let X = (X₁,...,Xₙ)ᵀ be a vector of random variables and A be an n×n matrix with det(A) ≠ 0 (so A is invertible). Define Y = AX.
The mapping Y = AX is automatically one-to-one (because A is invertible), and the inverse is X = A⁻¹Y. The Jacobian of the inverse transformation is simply A⁻¹, and its determinant is 1/det(A).
The joint p.d.f. of Y is:
fᵧ₁,...,ᵧₙ(y₁,...,yₙ) = fₓ₁,...,ₓₙ(x₁,...,xₙ) · |det(A⁻¹)| = (1/|det(A)|) · fₓ₁,...,ₓₙ(A⁻¹y)
where (x₁,...,xₙ) = A⁻¹(y₁,...,yₙ).
Linear transformations of random vectors appear everywhere in statistics:
Rotating or scaling multivariate normal distributions.
Principal component analysis (PCA) is a linear transformation.
Regression residuals are linear transformations of the response vector.
The formula above is the quickest route to the distribution of Y whenever the transformation is linear: compute A⁻¹, find det(A), substitute, done.
Discrete multivariate: fᵧ₁,...,ᵧₘ(y₁,...,yₘ) = Σ fₓ₁,...,ₓₙ(x₁,...,xₙ), summed over Aᵧ = {(x₁,...,xₙ) : u₁ = y₁,..., uₘ = yₘ}.
Continuous multivariate change of variables (one-to-one): fᵧ₁,...,ᵧₙ(y₁,...,yₙ) = fₓ₁,...,ₓₙ(s₁(y),...,sₙ(y)) · |det(J)| for y ∈ B.
Jacobian matrix (2×2 case): J = [[∂s₁/∂y₁, ∂s₁/∂y₂], [∂s₂/∂y₁, ∂s₂/∂y₂]].
Determinant (2×2): det([[a, b], [c, d]]) = ad − bc.
Linear transformation Y = AX: fᵧ(y) = (1/|det(A)|) · fₓ(A⁻¹y).
Students often forget to take the absolute value of the Jacobian determinant. The determinant itself can be negative (depending on the orientation of the transformation), but the density must be non-negative, so you always need |det(J)|.
A frequent error is trying to apply the multivariate change-of-variables formula when the number of output variables differs from the number of input variables (m ≠ n). The formula requires a square Jacobian. If m < n, you typically need to introduce auxiliary variables to make the dimensions match, then marginalise them out.
Students sometimes confuse the Jacobian of the forward transformation with the Jacobian of the inverse. The formula uses the Jacobian of the inverse (from y back to x). Using the forward Jacobian gives the reciprocal, which will produce a wrong answer.
For linear transformations, students occasionally compute det(A⁻¹) by inverting A first. This is unnecessary: det(A⁻¹) = 1/det(A), so you only need the determinant of A itself.
⚠️ Computing 2×2 Jacobians and their determinants is a bread-and-butter exam skill. You should be able to set up the Jacobian matrix, compute the determinant, and apply the formula quickly and without errors.
⚠️ The linear transformation result fᵧ(y) = (1/|det(A)|) · fₓ(A⁻¹y) is likely to appear in problems involving sums and differences of random variables, or affine transformations of multivariate normals.
⚠️ Watch for problems that give you a transformation from (X₁, X₂) to (Y₁, Y₂) where one of the Y variables is defined as an auxiliary (e.g. Y₂ = X₂). This is the standard trick for finding the marginal distribution of a single transformed variable: transform to a convenient joint, then integrate out the auxiliary.
⚠️ Always state the support B of the transformed variables explicitly. This is where most marks are lost on multivariate transformation problems.
Real-world applications: The multivariate change-of-variables formula is how the density of the multivariate normal is derived after a linear transformation. It also underpins the derivation of the Wishart distribution in multivariate analysis and appears in Bayesian inference when reparameterising models.
True or False: The multivariate change-of-variables formula can be applied when the number of output variables is less than the number of input variables.
Answer: False. You need equal dimensions (n = n). If m < n, introduce auxiliary variables first.
Fill in the blank: For a 2×2 matrix A = [[a, b], [c, d]], det(A) = ______.
Answer: ad − bc.
True or False: For a linear transformation Y = AX, the Jacobian determinant is |det(A)|.
Answer: False. The Jacobian of the inverse transformation is |det(A⁻¹)| = 1/|det(A)|.
Fill in the blank: In the multivariate change-of-variables formula, the Jacobian matrix J has entry Jᵢⱼ = ______.
Answer: ∂sᵢ/∂yⱼ (partial derivative of the i-th inverse function with respect to the j-th transformed variable).
True or False: If the forward transformation is one-to-one, the Jacobian determinant is always positive.
Answer: False. The determinant can be negative (the transformation may reverse orientation), which is why we take the absolute value.
Q: Let X₁ and X₂ be independent Exponential(1) random variables. Define Y₁ = X₁ + X₂ and Y₂ = X₁/(X₁ + X₂). Find the Jacobian of the inverse transformation.
A: The inverse is X₁ = Y₁Y₂ and X₂ = Y₁(1 − Y₂). The Jacobian matrix is J = [[∂X₁/∂Y₁, ∂X₁/∂Y₂], [∂X₂/∂Y₁, ∂X₂/∂Y₂]] = [[Y₂, Y₁], [1 − Y₂, −Y₁]]. So det(J) = Y₂(−Y₁) − Y₁(1 − Y₂) = −Y₁Y₂ − Y₁ + Y₁Y₂ = −Y₁. Therefore |det(J)| = Y₁.
Q: Let Y = AX where A = [[2, 1], [0, 3]] and (X₁, X₂) have joint density fₓ(x₁, x₂). Write the density of Y.
A: det(A) = 2(3) − 1(0) = 6. A⁻¹ = (1/6)[[3, −1], [0, 2]]. So fᵧ(y₁, y₂) = (1/6) · fₓ((3y₁ − y₂)/6, y₂/3).
Q: Explain why auxiliary variables are needed when transforming from n variables to fewer than n.
A: The change-of-variables formula requires a square Jacobian matrix (n×n). If you define only m < n new variables, you cannot form a square Jacobian. The standard approach is to define n − m auxiliary variables (often just setting Yₘ₊₁ = Xₘ₊₁, etc.) to make the system n×n, apply the formula, and then integrate out the auxiliary variables to get the marginal distribution of the m variables you care about.
The multivariate change-of-variables formula is the gateway to deriving the distributions of sums, ratios, and products of random variables, which appear throughout inferential statistics (e.g. deriving the t-distribution as a ratio of a normal and a chi-squared). Linear transformations connect directly to multivariate normal theory, where the key result is that any linear transformation of a multivariate normal is again multivariate normal. The Jacobian machinery also reappears in Bayesian statistics when changing parameterisations in posterior distributions.
Multivariate transformation, Jacobian, Jacobian determinant, change of variables multivariate, joint p.d.f. transformation, linear transformation random variables, matrix transformation density, auxiliary variable technique, marginalisation, functions of several random variables, NYU Analytical Statistics, determinant 2×2, inverse transformation multivariate, support of transformed variables, Y = AX density.