Difficulty: Intermediate | Prerequisites: Orthogonal/orthonormal bases, Gram-Schmidt process (Part 1 of these notes), subspaces, dimension
Part 1 of this lecture gave you the Gram-Schmidt process for building orthonormal bases. This second half puts that tool to work. The central result is that any vector in an inner product space can be uniquely split into a piece inside a subspace and a piece perpendicular to it. That perpendicular piece lives in the "orthogonal complement," and the splitting is done by orthogonal projection. This decomposition is the foundation for least-squares approximation, Fourier series, and a clean dimension-counting relationship between a subspace and its complement. If you take one thing from this lecture, it is Theorem 1 (the orthogonal decomposition) and the projection formula.
The orthogonal complement S⊥ collects every vector perpendicular to a set S, and it is always a subspace. For a finite-dimensional subspace W, every vector in V splits uniquely as x = w + z with w ∈ W and z ∈ W⊥. The component w = Proj_W(x) is the closest point in W to x. This gives V ≅ W ⊕ W⊥ and dim(W⊥) = dim(V) − dim(W).
Orthogonal complement (S⊥)
Given a nonempty subset S of an inner product space (V, ⟨·,·⟩), the orthogonal complement is S⊥ = {x ∈ V | ⟨x, s⟩ = 0 for all s ∈ S}. In simple terms, it is the collection of every vector in V that is perpendicular to everything in S.
Orthogonal projection (Proj_W)
Given a finite-dimensional subspace W with orthonormal basis {u₁, …, uₖ}, the orthogonal projection of x onto W is Proj_W(x) = ⟨x, u₁⟩u₁ + ⋯ + ⟨x, uₖ⟩uₖ. Think of it as the "shadow" of x dropped perpendicularly onto the subspace W.
Orthogonal decomposition
The unique way of writing any x ∈ V as x = w + z where w ∈ W and z ∈ W⊥. In simple terms, you split x into its component along W and its component perpendicular to W, and there is exactly one way to do this.
Direct sum (W ⊕ W⊥)
The statement V = W ⊕ W⊥ means every vector in V can be written uniquely as a sum of a vector in W and a vector in W⊥. The two subspaces together "fill" V without overlapping (except at the zero vector).
Best approximation
The vector in W that is closest to a given x ∈ V, measured by the norm ‖x − y‖. The best approximation is always Proj_W(x).
S⊥ = {x ∈ V | ⟨x, s⟩ = 0 for all s ∈ S}
Example in ℝ³. Let S = {(0, 0, 1)ᵀ} with the standard dot product. Then S⊥ is the set of all (x₁, x₂, x₃)ᵀ ∈ ℝ³ with x₃ = 0. Geometrically, S⊥ is the entire xy-plane.
Example in C⁰([−1,1]). Let S = {1} (the constant function) with ⟨f, g⟩ = ∫₋₁¹ f(x)g(x) dx. Then S⊥ is the set of all continuous functions f on [−1, 1] satisfying ∫₋₁¹ f(x) dx = 0. These are the functions whose average value on [−1, 1] is zero.
Lemma. S⊥ is always a subspace of V (even if S itself is not a subspace).
Proof sketch. Take v₁, v₂ ∈ S⊥ and c ∈ 𝔽. For any s ∈ S: ⟨v₁ + cv₂, s⟩ = ⟨v₁, s⟩ + c⟨v₂, s⟩ = 0 + 0 = 0. So v₁ + cv₂ ∈ S⊥.
Theorem. If the zero vector 0_V ∈ S, then S ∩ S⊥ = {0_V}.
Proof. If x ∈ S ∩ S⊥, then x ∈ S and ⟨x, y⟩ = 0 for all y ∈ S. Since x ∈ S, take y = x: ⟨x, x⟩ = 0, so x = 0_V.
Remark. If S = {0_V}, then S⊥ = V. Every vector is perpendicular to the zero vector.
Let W ⊂ (V, ⟨·,·⟩) be a finite-dimensional subspace. For each x ∈ V, there exist unique vectors w ∈ W and z ∈ W⊥ such that
x = w + z.
Moreover, if {u₁, …, uₖ} is an orthonormal basis for W, then
w = ⟨x, u₁⟩u₁ + ⋯ + ⟨x, uₖ⟩uₖ.
This means V is isomorphic to the direct sum W ⊕ W⊥.
(a) Existence: z = x − w ∈ W⊥.
Set w = Σᵢ₌₁ᵏ ⟨x, uᵢ⟩uᵢ and z = x − w. To show z ∈ W⊥, it suffices to show ⟨z, uᵢ⟩ = 0 for each basis vector uᵢ:
⟨z, uᵢ⟩ = ⟨x − Σⱼ ⟨x, uⱼ⟩uⱼ, uᵢ⟩ = ⟨x, uᵢ⟩ − Σⱼ ⟨x, uⱼ⟩⟨uⱼ, uᵢ⟩
Since {u₁, …, uₖ} is orthonormal, ⟨uⱼ, uᵢ⟩ = 0 for j ≠ i and = 1 for j = i. The sum collapses to ⟨x, uᵢ⟩ − ⟨x, uᵢ⟩ = 0.
(b) Uniqueness.
Suppose x = w + z = w̃ + z̃ with w, w̃ ∈ W and z, z̃ ∈ W⊥. Then w − w̃ = z̃ − z. The left side is in W and the right side is in W⊥, so both are in W ∩ W⊥ = {0_V}. Therefore w = w̃ and z = z̃.
The map Proj_W : V → W defined by
Proj_W(x) = Σᵢ₌₁ᵏ ⟨x, uᵢ⟩ uᵢ
is linear. Linearity follows from the linearity of the inner product in the first argument.
w = Proj_W(x) is the vector in W closest to x:
‖x − w‖ ≤ ‖x − y‖ for all y ∈ W.
Proof. Write x = w + z with z ∈ W⊥. For any y ∈ W:
‖x − y‖² = ‖(w − y) + z‖²
Since (w − y) ∈ W and z ∈ W⊥, the cross terms vanish:
= ‖w − y‖² + ‖z‖²
≥ ‖z‖² = ‖x − w‖².
Equality holds only when y = w.
For a finite-dimensional subspace W ⊂ V (with V also finite-dimensional):
dim(W⊥) = dim(V) − dim(W), which follows directly from V = W ⊕ W⊥.
The quotient map Q: V → V/W sending v ↦ [v] is linear with ker(Q) = W, so dim(W) + dim(V/W) = dim(V).
The restriction Q|_{W⊥} : W⊥ → V/W is an isomorphism, confirming W⊥ ≅ V/W.
Orthogonal complement:
S⊥ = {x ∈ V | ⟨x, s⟩ = 0, ∀s ∈ S}
Orthogonal projection (orthonormal basis {u₁, …, uₖ} for W):
Proj_W(x) = Σᵢ₌₁ᵏ ⟨x, uᵢ⟩ uᵢ
Best approximation inequality:
‖x − Proj_W(x)‖ ≤ ‖x − y‖ for all y ∈ W
Dimension formula:
dim(W⊥) = dim(V) − dim(W)
Exercise (end of lecture). In (C⁰([−1,1]), ∫₋₁¹ f(x)g(x) dx), let W = Pₙ(ℝ). Given f = sin x, the polynomial of degree n closest to sin x is Proj_{Pₙ}(sin x) = Σᵢ₌₁ⁿ⁺¹ (∫₋₁¹ sin(x) uᵢ(x) dx) uᵢ(x), where {u₁, …, uₙ₊₁} are the first n+1 normalised Legendre polynomials.
Orthogonal projection onto a subspace is exactly how least-squares regression works: given data that does not fit a model exactly, the best-fit parameters are those whose model output is the projection of the data vector onto the column space of the design matrix. The best approximation theorem also underpins signal processing, where you approximate a signal (a function) by projecting it onto a finite-dimensional subspace of basis functions (sines, cosines, wavelets, or polynomials).
Students often think S⊥ depends on what S "looks like" as a set, but S⊥ only depends on Span(S). Two different sets with the same span have the same orthogonal complement.
A frequent error is assuming S⊥⊥ = S. This is true when S is a subspace of a finite-dimensional space, but not in general. If S is just an arbitrary set of vectors, S⊥⊥ = Span(S) (the closure to a subspace).
Students sometimes confuse the projection formula for an orthogonal basis with the formula for an orthonormal basis. With an orthonormal basis, the denominator ‖uᵢ‖² = 1 disappears. With a merely orthogonal basis, you must include the ‖vᵢ‖² denominators.
The best approximation theorem requires W to be a subspace, not just any subset. Projecting onto an arbitrary set is a different (and generally harder) problem.
⚠️ The projection formula Proj_W(x) = Σ ⟨x, uᵢ⟩uᵢ is one of the most-tested formulas in the course. Know it for both orthogonal and orthonormal bases.
⚠️ The uniqueness proof for the decomposition x = w + z is a classic short exam proof. The key step is showing w − w̃ ∈ W ∩ W⊥ = {0}.
⚠️ Be prepared to compute Proj_W(x) for a concrete subspace of ℝⁿ: find or be given an orthonormal basis for W, then apply the formula.
⚠️ The dimension formula dim(W⊥) = dim(V) − dim(W) is frequently tested as a quick-answer or true/false question.
⚠️ The Legendre polynomial exercise at the end of the lecture (approximating sin x by a polynomial via projection) is a strong candidate for a homework or exam problem.
True or False: The orthogonal complement of a subspace is always a subspace.
A: True. S⊥ is a subspace regardless of whether S itself is a subspace.
Fill in the blank: If W is a 3-dimensional subspace of ℝ⁷, then dim(W⊥) = ______.
A: 4.
True or False: If x ∈ W, then Proj_W(x) = x.
A: True. The component of x perpendicular to W is zero, so x = x + 0.
True or False: The best approximation to x from W can sometimes be a vector y ∈ W with y ≠ Proj_W(x).
A: False. The projection is always the unique closest point.
Q: Let W = Span{(1, 0, 0)ᵀ, (0, 1, 0)ᵀ} in ℝ³ with the standard dot product. Find W⊥.
A: W is the xy-plane. A vector (x₁, x₂, x₃)ᵀ is in W⊥ iff it is perpendicular to both (1,0,0)ᵀ and (0,1,0)ᵀ, which forces x₁ = 0 and x₂ = 0. So W⊥ = Span{(0, 0, 1)ᵀ}, the z-axis.
Q: Prove that S ∩ S⊥ = {0} when S is a subspace.
A: Let x ∈ S ∩ S⊥. Then x ∈ S and ⟨x, y⟩ = 0 for all y ∈ S. Since x ∈ S, we may take y = x, giving ⟨x, x⟩ = 0. By positive-definiteness, x = 0.
Q: Let W = Span{(1/√2, 1/√2, 0)ᵀ} in ℝ³. Compute Proj_W((3, 1, 5)ᵀ).
A: The given basis vector u₁ = (1/√2, 1/√2, 0)ᵀ is already unit-length. Proj_W(x) = ⟨x, u₁⟩u₁ = (3/√2 + 1/√2)(1/√2, 1/√2, 0)ᵀ = (4/√2)(1/√2, 1/√2, 0)ᵀ = (2, 2, 0)ᵀ.
Q: Why does the proof of the best approximation theorem rely on z ∈ W⊥?
A: Because the cross terms ⟨w − y, z⟩ and ⟨z, w − y⟩ vanish when z ∈ W⊥ and w − y ∈ W. This lets you split ‖x − y‖² into ‖w − y‖² + ‖z‖², which is clearly ≥ ‖z‖² = ‖x − w‖².
Q: In the exercise, what is the role of the Legendre polynomials when approximating sin x by a degree-n polynomial on [−1, 1]?
A: The normalised Legendre polynomials form an orthonormal basis for Pₙ(ℝ) with respect to ⟨f, g⟩ = ∫₋₁¹ fg dx. You compute the projection of sin x onto Pₙ(ℝ) using this basis, and the result is the polynomial of degree ≤ n that minimises ‖sin x − p‖ in the L² norm on [−1, 1].
The orthogonal decomposition V = W ⊕ W⊥ connects to the rank-nullity theorem: the quotient space V/W is isomorphic to W⊥, giving another way to see that dim(ker T) + dim(im T) = dim(V) for a linear map T. The best approximation theorem is the theoretical engine behind least-squares solutions to Ax = b when b is not in the column space of A, a topic covered in applied linear algebra courses. The Legendre polynomial exercise at the end of this lecture bridges to Fourier analysis, where you project functions onto orthonormal bases of trigonometric functions rather than polynomials.
orthogonal complement, orthogonal projection, Proj_W, best approximation theorem, closest vector in subspace, orthogonal decomposition, direct sum, W perp, perpendicular complement, least squares, dimension formula orthogonal complement, uniqueness of decomposition, Legendre polynomial approximation, L2 projection, MATH 416, abstract linear algebra, UIUC