Hilbert Spaces 7 | Approximation Formula

TL;DR

Orthogonality in an inner product space (⟨X,Y⟩=0) yields the generalized Pythagorean theorem: ‖X+Y‖² = ‖X‖² + ‖Y‖².

Briefing Cornell Notes

Briefing

Hilbert spaces make a familiar geometric idea—“the closest point in a set”—work cleanly in infinite-dimensional settings, but only when two conditions are met: completeness (the space is a Hilbert space) and a strong shape constraint on the target set (it must be closed and convex). The payoff is a theorem guaranteeing that every vector has a single “best approximation” inside such a set, meaning there is one and only one point in the set that minimizes the distance to the given vector.

The groundwork starts with the geometry of inner product spaces. When two vectors X and Y are orthogonal—captured by the inner product ⟨X,Y⟩ equaling zero—one can form a right-angle triangle in the abstract vector space. The hypotenuse corresponds to X+Y, and using the norm induced by the inner product (‖v‖ = √⟨v,v⟩), the squared length of X+Y expands via bilinearity into four inner-product terms. The mixed “cross” terms vanish precisely because of orthogonality, leaving the generalized Pythagorean theorem: ‖X+Y‖² = ‖X‖² + ‖Y‖². This establishes that even in high (or infinite) dimensions, orthogonality still supports reliable distance calculations.

From there, the discussion shifts to approximation. In linear algebra, orthogonal projections are often used to find the closest point to a vector from a subspace. In a Hilbert space, the same goal becomes: given a set U and a vector X, can one find an element u in U that minimizes ‖X−u‖? While the distance between X and U can always be defined using an infimum over all possible values of ‖X−u‖, the existence of an actual minimizer is not automatic—especially in infinite-dimensional spaces.

The key theorem (set up here, with proof deferred to the next installment) requires that the ambient space be complete and that U be closed and convex. Closedness matters because limits of minimizing sequences must stay in U; convexity matters because straight-line segments between points in U must remain inside U, preventing “holes” or disconnected shapes that could break uniqueness or even existence. Every subspace automatically satisfies convexity, which is why projection-style approximation behaves well for subspaces.

Under these assumptions, the theorem promises: for every vector X, there exists a unique best approximation in U, often denoted as X restricted to U. This best approximating point lies in U and achieves the infimum distance, so the distance from X to U equals ‖X − (best approximation)‖. The next step—coming in the following video—is to connect this best approximation to orthogonality, turning the geometric intuition into a precise characterization.

Cornell Notes

The generalized Pythagorean theorem in inner product spaces shows that orthogonality (⟨X,Y⟩=0) makes distance computations behave like right triangles: ‖X+Y‖² = ‖X‖² + ‖Y‖². The approximation problem asks whether, for a given set U and vector X, there is an element u∈U that minimizes ‖X−u‖. Distance can always be defined using an infimum over all u∈U, but a true minimizer may fail to exist without extra structure. The approximation theorem requires the ambient space to be a Hilbert space (complete inner product space) and the set U to be closed and convex. Then every X has a unique best approximation in U, and the minimal distance equals the norm of X minus that point.

Why does orthogonality lead to a Pythagorean-type formula in inner product spaces?

Orthogonality means ⟨X,Y⟩=0. For the hypotenuse vector X+Y, the squared norm is ‖X+Y‖² = ⟨X+Y, X+Y⟩. Expanding using additivity/bilinearity gives four terms: ⟨X,X⟩ + ⟨X,Y⟩ + ⟨Y,X⟩ + ⟨Y,Y⟩. The mixed terms vanish because ⟨X,Y⟩=0 (and likewise ⟨Y,X⟩=0), leaving ‖X‖² + ‖Y‖².

What is the approximation problem, and how is “distance to a set” defined when a minimizer might not exist?

Given a set U and a vector X, the goal is to find u∈U minimizing ‖X−u‖. Even if no u achieves the minimum, one can still define the distance using the infimum: consider all values ‖X−u‖ as u ranges over U, take their greatest lower bound (infimum), and call that the distance. This guarantees a well-defined number, but not necessarily a point u that attains it.

Why do completeness and convexity both matter for the existence of a best approximation?

Completeness (being a Hilbert space) helps ensure that minimizing sequences have limits that remain meaningful in the space; without completeness, limits used to produce a minimizer can fall outside the space. Convexity ensures the set has no “indentations” that could prevent a closest point from existing or could destroy uniqueness. Convexity means that for any two points u,v∈U, every point on the straight line segment between them also lies in U.

How do closedness and convexity interact with the idea of minimizing ‖X−u‖?

Closedness ensures that if a sequence of points u_n∈U approaches a limit, that limit point still belongs to U. This is crucial when constructing a minimizer via limits of near-best approximations. Convexity restricts the geometry so that the closest-point behavior is stable: straight-line segments between candidate points stay inside U, which supports uniqueness of the minimizer.

What does the theorem guarantee once the assumptions are satisfied?

For every vector X, there exists a unique best approximation point in U, denoted informally as X restricted to U. This point lies in U and minimizes the distance, so ‖X − (best approximation)‖ equals the distance from X to U (the infimum over all u∈U).

Review Questions

What exact algebraic step makes the cross terms disappear in the generalized Pythagorean theorem?
How does the infimum-based definition of distance differ from the existence of a minimizer?
Which two properties of U are required for a unique best approximation, and why is convexity singled out?

Key Points

1
Orthogonality in an inner product space (⟨X,Y⟩=0) yields the generalized Pythagorean theorem: ‖X+Y‖² = ‖X‖² + ‖Y‖².
2
Distance from a vector X to a set U can always be defined via an infimum over ‖X−u‖, even if no minimizing u exists.
3
A Hilbert space’s completeness is essential for turning an infimum distance into an actual best approximation point.
4
The target set U must be closed and convex to guarantee existence and uniqueness of the best approximation.
5
Convexity means every line segment between two points in U stays entirely inside U; subspaces automatically satisfy this.
6
When the assumptions hold, there is exactly one point in U minimizing the distance to X, and the minimal distance equals the norm of X minus that point.

Highlights

Orthogonality makes the “mixed” inner-product terms vanish, producing a Pythagorean theorem for vectors: ‖X+Y‖² = ‖X‖² + ‖Y‖².

Even when distance to a set is well-defined using an infimum, a closest point may not exist without extra assumptions.

The best-approximation theorem requires both Hilbert-space completeness and that U be closed and convex to ensure existence and uniqueness.

Convexity is the geometric safeguard: straight-line segments between points in U cannot leave the set. 

Topics

Hilbert Spaces
Approximation Formula
Orthogonality
Pythagorean Theorem
Best Approximation