SVD Singular Value Decomposition Reference Cheat Sheet

Singular value decomposition is a matrix factorization used throughout linear algebra, data science, statistics, engineering, and numerical computing. This cheat sheet helps students connect the formula $A = U\Sigma V^T$ to geometry, rank, least squares, compression, and stability. It is especially useful because SVD works for any real $m \times n$ matrix, including non-square and singular matrices.

The core idea is that a matrix transformation can be broken into a rotation or reflection, a coordinate-wise scaling, and another rotation or reflection. The singular values $\sigma_i$ measure the strength of the transformation along special orthogonal directions. The columns of $V$ are right singular vectors, the columns of $U$ are left singular vectors, and the diagonal entries of $\Sigma$ are nonnegative singular values.

Key Facts

Every real matrix $A \in \mathbb{R}^{m \times n}$ has an SVD $A = U\Sigma V^T$ , where $U$ and $V$ have orthonormal columns and $\Sigma$ is diagonal or rectangular diagonal.
The singular values satisfy $\sigma_i \ge 0$ and are usually ordered as $\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_r > 0$ .
The right singular vectors are eigenvectors of $A^T A$ , and $A^T A v_i = \sigma_i^2 v_i$ .
The left singular vectors are related by $u_i = \frac{A v_i}{\sigma_i}$ for every nonzero singular value $\sigma_i$ .
The rank of $A$ equals the number of nonzero singular values, so $\operatorname{rank}(A) = r$ .
The spectral norm of $A$ is $\|A\|_2 = \sigma_1$ , and the Frobenius norm is $\|A\|_F = \sqrt{\sum_i \sigma_i^2}$ .
The Moore-Penrose pseudoinverse from the SVD is $A^+ = V\Sigma^+ U^T$ , where each nonzero $\sigma_i$ becomes $\frac{1}{\sigma_i}$ .
The best rank- $k$ approximation in the $2$ -norm or Frobenius norm is $A_k = \sum_{i=1}^{k} \sigma_i u_i v_i^T$ .

Vocabulary

Singular Value Decomposition: A factorization $A = U\Sigma V^T$ that expresses a matrix as orthogonal directions, nonnegative scalings, and orthogonal directions.
Singular Value: A nonnegative number $\sigma_i$ on the diagonal of $\Sigma$ that measures how much $A$ stretches a corresponding singular vector direction.
Right Singular Vector: A column vector $v_i$ of $V$ that is an eigenvector of $A^T A$ and represents an input direction for the transformation.
Left Singular Vector: A column vector $u_i$ of $U$ that gives the output direction produced by $A v_i = \sigma_i u_i$ .
Pseudoinverse: The matrix $A^+ = V\Sigma^+ U^T$ that generalizes the inverse for rectangular or singular matrices and is used in least squares problems.
Low-Rank Approximation: An approximation $A_k$ that keeps only the largest $k$ singular values and their singular vectors to capture the strongest structure in $A$ .

Common Mistakes to Avoid

Confusing eigenvalues with singular values is wrong because singular values are always nonnegative and come from $\sqrt{\lambda_i(A^T A)}$ , not necessarily from eigenvalues of $A$ .
Forgetting the transpose in $A = U\Sigma V^T$ is wrong because the right singular vectors must act through $V^T$ , not through $V$ .
Assuming SVD requires a square matrix is wrong because SVD exists for every $m \times n$ matrix, including rectangular matrices.
Inverting zero singular values in $\Sigma^+$ is wrong because only nonzero singular values become $\frac{1}{\sigma_i}$ , while zeros stay zero.
Keeping the smallest singular values for compression is wrong because the largest singular values contain the strongest action and minimize approximation error.

Practice Questions

1 A matrix has singular values $9$ , $4$ , $1$ , and $0$ . Find $\operatorname{rank}(A)$ , $\|A\|_2$ , and $\|A\|_F$ .
2 If $A = U\Sigma V^T$ and $\Sigma$ has nonzero diagonal entries $5$ , $2$ , and $\frac{1}{2}$ , what are the corresponding nonzero diagonal entries of $\Sigma^+$ ?
3 For singular values $12$ , $6$ , $3$ , and $1$ , write the Frobenius norm error $\|A - A_2\|_F$ for the best rank- $2$ approximation.
4 Explain why SVD can solve least squares problems even when $A^T A$ is singular or poorly conditioned.

Understanding SVD Singular Value Decomposition Reference

A useful way to read an SVD is as a list of independent input-output channels. Start with one right singular direction. A vector pointing exactly in that direction is sent to the matching left singular direction, with its length multiplied by the matching singular value.

Directions with a zero singular value disappear completely under the matrix transformation. Those lost directions form the null space. This explains why some systems have no unique solution.

Different input vectors can produce the same output when their difference lies in the null space. The nonzero left singular directions describe the column space, meaning every output the matrix can actually create.

Singular values reveal more than rank. Their sizes show how sensitive a calculation is to small errors. If the largest singular value is much bigger than the smallest nonzero one, the matrix is ill-conditioned.

A tiny measurement error in an output can then cause a large error when trying to recover an input. This happens because solving a system requires dividing by singular values. Dividing by a very small value magnifies noise.

In numerical work, a value may be treated as zero when it is too small compared with the largest value. This is a practical decision based on machine precision and the accuracy of the data, not just a theorem about exact numbers.

The pseudoinverse is especially important for data that give too many equations or too few. With more equations than unknowns, an exact solution may not exist because measurements disagree slightly. The pseudoinverse chooses an input whose predicted output is as close as possible to the measured output in ordinary distance.

With fewer equations than unknowns, there may be many exact solutions. It selects the one with the smallest length. These choices are useful in fitting a line to noisy data, estimating forces from sensor readings, reconstructing images, and training simple linear models.

Small singular values can make these answers unstable, so analysts often use truncated SVD. They keep only reliable large directions and discard weak directions that mostly amplify noise.

Low-rank approximation works because many real data sets contain repeated patterns. A grayscale image can be stored as a matrix of brightness values. The first few singular components often capture broad lighting, large shapes, and major edges.

Later components add fine detail. Keeping only some components reduces storage, though it removes information. The same idea appears in recommendation systems, document analysis, genetics, and signal processing.

When studying SVD, pay attention to dimensions at every step. Track which vectors live in the input space and which live in the output space.

Remember that singular vectors are not always unique, especially when singular values repeat. What remains meaningful is the subspace spanned by those vectors and the amount of stretching represented by the singular values.

Sign in to save

Sign in to save

SVD Singular Value Decomposition Reference Cheat Sheet

Related Tools

Related Labs

Related Worksheets

Related Infographics

Study as Flashcards

Key Facts

Vocabulary

Common Mistakes to Avoid

Practice Questions

Understanding SVD Singular Value Decomposition Reference