Difficulty: Intermediate-Advanced | Prerequisites: Inner product spaces, orthonormal bases (Gram-Schmidt), subspaces, dimension and rank.
Every vector in an inner product space can be split uniquely into a piece inside a finite-dimensional subspace W and a piece orthogonal to W. The piece inside W is the orthogonal projection, computed by taking inner products with an orthonormal basis of W. That projection is also the closest vector in W to the original, which is the theoretical backbone of least squares problems.
Orthogonal complement (S⊥)
The set of all vectors in V that are orthogonal to every vector in S: S⊥ = { v ∈ V | ⟨v, s⟩ = 0 for all s ∈ S }. It is always a subspace of V, even when S itself is not.
In simple terms, S⊥ is everything in the space that is "perpendicular" to S.
Orthogonal projection (Proj_W(v))
The unique vector w in W such that v − w is orthogonal to every vector in W. If {u₁, …, uₖ} is an orthonormal basis for W, then Proj_W(v) = ⟨v, u₁⟩u₁ + … + ⟨v, uₖ⟩uₖ.
Think of it as the "shadow" of v dropped straight down onto the subspace W.
Orthogonal decomposition
The fact that every v ∈ V can be written uniquely as v = w + z where w ∈ W and z ∈ W⊥. This splits the whole space into two perpendicular pieces.
Direct sum (W ⊕ W⊥)
The space of all pairs (w, z) with w ∈ W and z ∈ W⊥. Theorem 1 says V is isomorphic to W ⊕ W⊥ when W is finite-dimensional.
In simple terms, the space breaks cleanly into W and "everything perpendicular to W," with no overlap except the zero vector.
Adjoint / conjugate transpose (A)*
For a matrix A, the adjoint A* is the conjugate transpose: A* = (Ā)ᵀ = (Aᵀ)̄. Over the reals, this is just the ordinary transpose Aᵀ. It satisfies ⟨Ax, y⟩ = ⟨x, A*y⟩.
Think of it as the matrix that "moves A to the other side" of an inner product.
For any subset S of an inner product space (V, ⟨ , ⟩), the orthogonal complement S⊥ is always a subspace of V.
The zero vector 0_V belongs to S⊥ (since ⟨0, s⟩ = 0 for all s).
If S is itself a subspace, then S ∩ S⊥ = {0_V}. The only vector that is both in a subspace and perpendicular to it is zero.
Let W ⊂ (V, ⟨ , ⟩) be a finite-dimensional subspace. Then each v ∈ V can be written uniquely as:
v = w + z, where w ∈ W and z ∈ W⊥
Moreover, if {u₁, …, uₖ} is an orthonormal basis of W, then:
w = ⟨v, u₁⟩u₁ + … + ⟨v, uₖ⟩uₖ = Proj_W(v)
Proof sketch:
Define w = Σ ⟨v, uᵢ⟩uᵢ and z = v − w.
Need to show z ∈ W⊥, i.e. ⟨z, y⟩ = 0 for all y ∈ W.
Since {u₁, …, uₖ} spans W, it suffices to check ⟨z, uⱼ⟩ = 0 for each basis vector uⱼ.
Expanding: ⟨v − Σ⟨v, uᵢ⟩uᵢ, uⱼ⟩ = ⟨v, uⱼ⟩ − Σ⟨v, uᵢ⟩⟨uᵢ, uⱼ⟩ = ⟨v, uⱼ⟩ − ⟨v, uⱼ⟩ = 0, using orthonormality (⟨uᵢ, uⱼ⟩ = δᵢⱼ).
For uniqueness: if v = w + z = ŵ + ẑ, then w − ŵ = ẑ − z. The left side is in W, the right in W⊥. Since W ∩ W⊥ = {0}, both sides are zero, so w = ŵ and z = ẑ.
Corollary 1: Proj_W is well-defined
The map Proj_W : V → W does not depend on which orthonormal basis {u₁, …, uₖ} you choose for W. The uniqueness of w in the decomposition v = w + z guarantees this.
Corollary 2: V ≅ W ⊕ W⊥
The map T : W ⊕ W⊥ → V defined by (w, z) ↦ w + z is a linear isomorphism. Theorem 1 gives surjectivity (every v = w + z) and injectivity (the decomposition is unique).
Corollary 3: W⊥ ≅ V/W and dim(W⊥) = dim(V) − dim(W)
By the rank-nullity theorem applied to the quotient map Q: V = W ⊕ W⊥ → V/W, we get dim(V) = dim(W) + dim(V/W). From Corollary 2, dim(V) = dim(W) + dim(W⊥). So dim(W⊥) = dim(V) − dim(W), and W⊥ is isomorphic to the quotient space V/W.
w = Proj_W(v) is the vector in W closest to v. That is:
‖v − w‖ ≤ ‖v − y‖ for all y ∈ W
Proof sketch:
Write v = w + z with w ∈ W and z ∈ W⊥. For any y ∈ W:
‖v − y‖² = ‖(w + z) − y‖² = ‖(w − y) + z‖²
Since (w − y) ∈ W and z ∈ W⊥, the cross terms vanish: ⟨w − y, z⟩ = 0.
So ‖v − y‖² = ‖w − y‖² + ‖z‖² ≥ ‖z‖² = ‖v − w‖².
Equality holds only when y = w, confirming that the projection is the unique closest point.
The key insight here is that the Pythagorean theorem applies because the two components are orthogonal. The "error" z = v − w is irreducible: no matter which y ∈ W you pick, you cannot make the distance smaller than ‖z‖.
Orthogonal complement
S⊥ = { v ∈ V | ⟨v, s⟩ = 0 ∀ s ∈ S }
Orthogonal projection onto W (given orthonormal basis {u₁, …, uₖ})
Proj_W(v) = ⟨v, u₁⟩u₁ + ⟨v, u₂⟩u₂ + … + ⟨v, uₖ⟩uₖ = Σᵢ₌₁ᵏ ⟨v, uᵢ⟩uᵢ
Dimension formula
dim(W⊥) = dim(V) − dim(W)
Geometric picture
Imagine W as a plane through the origin and W⊥ as the line perpendicular to it. Any vector v is the diagonal of a rectangle whose sides lie along W (the projection w) and W⊥ (the residual z). The projection is the foot of the perpendicular from v to W.
Orthogonal projection is the mathematical engine behind signal processing (decomposing a signal into frequency components), computer graphics (projecting 3D scenes onto 2D screens), and statistics (linear regression finds the projection of the data vector onto the column space of the design matrix). Anywhere you need the "closest fit" of one thing by another in a least-squares sense, you are using Theorem 2.
Students often assume orthogonal projection depends on which orthonormal basis you pick for W. It does not. Corollary 1 guarantees the result is the same for any orthonormal basis.
A common error is forgetting the basis must be orthonormal. If you use an orthogonal but not normalised basis, the formula Σ ⟨v, uᵢ⟩uᵢ is wrong. You either normalise first or use the modified formula Σ (⟨v, uᵢ⟩ / ⟨uᵢ, uᵢ⟩) uᵢ.
Students sometimes think S⊥ depends on S being a subspace. It does not: S⊥ is defined for any subset S, and S⊥ is always a subspace regardless.
Confusing "v is orthogonal to w" with "v is in W⊥." The statement v ∈ W⊥ means v is orthogonal to every vector in W, not just one particular w.
⚠️ The projection formula Proj_W(v) = Σ ⟨v, uᵢ⟩uᵢ only works with an orthonormal basis. If they hand you a non-orthonormal basis, run Gram-Schmidt first.
⚠️ Uniqueness of the decomposition v = w + z is a favourite proof question. Know the W ∩ W⊥ = {0} argument.
⚠️ The best-approximation proof (Theorem 2) is short and elegant. It relies on the Pythagorean identity for orthogonal vectors. Expect to reproduce it on an exam.
⚠️ Corollary 3 (dim W⊥ = dim V − dim W) often appears as a "compute the dimension" problem. Given dim V and dim W, you can immediately read off dim W⊥.
True or False: S⊥ is a subspace of V even when S is not a subspace. (True)
Fill in the blank: If W is a 3-dimensional subspace of R⁷, then dim(W⊥) = ___. (4)
True or False: The orthogonal projection Proj_W(v) depends on the choice of orthonormal basis for W. (False)
True or False: If v ∈ W, then Proj_W(v) = v. (True)
Fill in the blank: The vector in W closest to v is ___. (Proj_W(v))
Q: Let W be a subspace of an inner product space V with orthonormal basis {u₁, u₂}. Write the formula for Proj_W(v) and state what property makes v − Proj_W(v) special.
A: Proj_W(v) = ⟨v, u₁⟩u₁ + ⟨v, u₂⟩u₂. The residual v − Proj_W(v) lies in W⊥, meaning it is orthogonal to every vector in W.
Q: Prove that the decomposition v = w + z (w ∈ W, z ∈ W⊥) is unique.
A: Suppose v = w + z = ŵ + ẑ. Then w − ŵ = ẑ − z. The left side is in W, the right is in W⊥. Since W ∩ W⊥ = {0_V}, both sides equal zero, so w = ŵ and z = ẑ.
Q: Why does the best-approximation proof (Theorem 2) rely on the Pythagorean theorem?
A: Because (w − y) ∈ W and z ∈ W⊥ are orthogonal, we get ‖(w − y) + z‖² = ‖w − y‖² + ‖z‖². This decomposition of the squared distance is what shows ‖v − y‖² ≥ ‖z‖² = ‖v − w‖² for any y ∈ W.
Q: If V = R⁴ and W = span{(1,0,0,0), (0,1,0,0)}, describe W⊥ and find dim(W⊥).
A: W⊥ = span{(0,0,1,0), (0,0,0,1)}, which is all vectors whose first two coordinates are zero. dim(W⊥) = 4 − 2 = 2.
This material connects directly to Gram-Schmidt orthogonalisation, which is the tool you use to produce the orthonormal basis the projection formula requires. It also leads into the next major topic: least squares approximation, where Theorem 2 is applied to the column space of a matrix to solve over-determined systems Ax = b. The dimension formula dim(W⊥) = dim(V) − dim(W) is a restatement of rank-nullity in the inner product setting.
orthogonal complement, orthogonal projection, Proj_W, best approximation theorem, closest point, orthogonal decomposition, direct sum, W perp, perpendicular subspace, inner product space, Pythagorean theorem for inner products, finite-dimensional subspace, rank-nullity, Gram-Schmidt, quotient space V/W, MATH 416, abstract linear algebra