Abstract Linear Algebra 49 | Singular Value Decomposition (Overview)

TL;DR

Singular value decomposition expresses any m×n matrix A as A = U Σ V*, with U and V unitary and Σ diagonal in the rectangular sense.

Briefing Cornell Notes

Briefing

Singular value decomposition (SVD) turns any matrix—square or rectangular—into a “diagonal” core using unitary changes of basis, making the matrix’s action easy to interpret and compute with. The key move is to relax the strict relationship used in earlier decompositions: instead of requiring the left and right factors to be inverses of each other (as in diagonalization or Jordan form), SVD keeps them unitary but allows them to be unrelated. That tradeoff is what makes it possible to force the middle matrix to be diagonal (in the rectangular sense) for essentially every matrix.

Earlier matrix decompositions split a matrix A into three parts: a left factor U, a middle factor D (diagonal or Jordan form), and a right factor V, with strong constraints tying U and V together. In the unitary-friendly Schur decomposition, U and V are unitary and inverses of each other, but the middle matrix is only upper triangular, not diagonal. SVD targets the missing ingredient: a diagonal middle matrix. Achieving that requires weakening the connection between the unitary factors—U and V no longer have to be inverses—so A and the diagonal core are not similar. Still, they remain equivalent: the diagonal core represents the same linear map under different choices of bases.

Concretely, for an m×n matrix A, SVD takes the form A = U Σ V*, where U is an m×m unitary matrix, V* is the conjugate transpose of an n×n unitary matrix V (so V* = V^{-1}), and Σ is an m×n diagonal-like matrix. “Diagonal” here means that if the rectangular matrix is cut down to an ordinary square, the result is a standard diagonal matrix: entries on the diagonal are the singular values, while everything below (or to the right, depending on whether there are more columns than rows) is zero. This definition is what lets SVD handle rectangular matrices without special cases.

Because U and V are unitary, they preserve lengths and angles, so the change of basis is geometrically well-behaved. The decomposition also has a clean linear-map interpretation. View A as the matrix representation of a linear map L: C^n → C^m with respect to the standard bases. Then Σ is the matrix representation of the same map L but with respect to new orthonormal bases in the input and output spaces. Those new bases are encoded in the columns of V (for the input side) and U (for the output side). The central goal becomes: choose orthonormal bases so that the representation of L becomes diagonal.

The transcript frames the remaining task as constructing the specific unitary matrices U and V that achieve this diagonal form. It points to the spectral theorem for normal matrices as the tool that guarantees such unitary choices, setting up the explicit calculations for the next step in the series.

Cornell Notes

Singular value decomposition writes any m×n matrix A as A = U Σ V*, where U and V are unitary and Σ is diagonal in the rectangular sense. The diagonalization is achieved by relaxing the strict inverse relationship between the unitary factors used in earlier decompositions: A and Σ are not similar, but they are equivalent, meaning they represent the same linear map under different bases. Unitarity matters because it preserves lengths and angles, so the basis changes are orthonormal. Interpreting A as a linear map L: C^n → C^m, the columns of V give the new input basis and the columns of U give the new output basis, making Σ the representation of L in those bases. The next step is computing U and V explicitly, with the spectral theorem for normal matrices providing the construction route.

Why does SVD relax the relationship between the unitary factors compared with Schur or diagonal/Jordan forms?

Schur decomposition keeps the left and right unitary factors tied together (they are inverses), but the middle matrix is only upper triangular. SVD wants the middle matrix to be diagonal (rectangular diagonal). To force that stronger structure, the unitary matrices U and V are allowed to be unitary without being inverses of each other. That breaks similarity—A and Σ are not similar—but it preserves equivalence, so they still represent the same linear map under different bases.

What does “diagonal” mean for a rectangular matrix in the SVD formula A = U Σ V*?

For an m×n matrix, Σ is arranged so that the diagonal entries are singular values and the rest is zero in the appropriate triangular region. If the rectangular matrix is cut down to an ordinary square, the result is a standard diagonal matrix. If m ≥ n, the nonzero structure sits in the top-left n×n diagonal block with zeros below; if n ≥ m, the diagonal block extends across the top-left m×m with zeros to the right.

How does unitarity (U and V unitary) connect to geometry and computation?

Unitary transformations preserve lengths and angles, so the basis changes encoded by U and V* are orthonormal changes of coordinates. This makes the decomposition stable and interpretable: the linear map is expressed in orthonormal bases where its action becomes diagonal-like. Also, unitary matrices are easy to invert: V* = V^{-1} (and similarly for U), which supports the equivalence interpretation.

In what sense are A and Σ “equivalent” even though they are not similar?

Similarity would require a single change of basis that transforms A into Σ via conjugation. SVD instead uses different basis changes on the input and output spaces: one unitary basis change on C^n and another on C^m. That means A and Σ represent the same abstract linear map L: C^n → C^m, but with respect to different orthonormal bases on domain and codomain.

How do the columns of U and V relate to the new bases for the linear map L: C^n → C^m?

The columns of V (inside V) form an orthonormal basis for the input space C^n, while the columns of U form an orthonormal basis for the output space C^m. In the decomposition A = U Σ V*, the V* factor corresponds to changing coordinates in the domain, Σ acts in the diagonalized coordinate system, and U changes coordinates in the codomain. Thus, V determines the new input basis and U determines the new output basis.

What role does the spectral theorem for normal matrices play in obtaining U and V?

The transcript indicates that constructing the unitary matrices U and V that achieve the diagonal form relies on the spectral theorem for normal matrices. That theorem guarantees the existence of unitary diagonalization for the relevant normal matrices arising in the SVD construction, providing the mechanism to build the orthonormal bases encoded by U and V.

Review Questions

How does SVD achieve a diagonal middle matrix when Schur decomposition only guarantees an upper triangular one?
Explain why A and Σ are equivalent but not necessarily similar in the SVD framework.
For an m×n matrix, describe where the nonzero entries of Σ can appear and how this matches the “cut to a square” definition of diagonal.

Key Points

1
Singular value decomposition expresses any m×n matrix A as A = U Σ V*, with U and V unitary and Σ diagonal in the rectangular sense.
2
SVD forces a diagonal middle matrix by relaxing the strict inverse relationship between the unitary factors used in earlier decompositions, so A and Σ are not similar.
3
Despite not being similar, A and Σ are equivalent because they represent the same linear map under different orthonormal bases on the domain and codomain.
4
Unitarity preserves lengths and angles, making the basis changes geometrically meaningful and numerically well-behaved.
5
For rectangular matrices, “diagonal” means the nonzero singular values lie on a diagonal block, with zeros filling the remaining triangular region (below or to the right depending on dimensions).
6
The columns of V define the new orthonormal basis in C^n (the input space), while the columns of U define the new orthonormal basis in C^m (the output space).
7
Constructing U and V explicitly is tied to the spectral theorem for normal matrices, which provides the needed unitary structure.

Highlights

SVD is built to produce a diagonal core Σ for any matrix, including rectangular ones, by allowing the unitary factors to be unrelated by inversion.

The decomposition preserves geometry: unitary changes of basis keep lengths and angles intact while turning the linear map into a diagonal-like form.

“Diagonal” for rectangular Σ is defined so that cutting it to a square yields an ordinary diagonal matrix.

Even without similarity, SVD still captures the same abstract linear map by using different basis changes on input and output spaces.

Topics

Singular Value Decomposition
Unitary Matrices
Rectangular Diagonal Form
Equivalence of Linear Maps
Spectral Theorem