Principal Component Analysis Reference Cheat Sheet

Key Facts

• For a data matrix $X$ with $n$ observations and $p$ variables, the centered data matrix is $X_c = X - \bar{X}$, where each column mean is subtracted.
• The sample covariance matrix for centered data is $S = \frac{1}{n-1}X_c^T X_c$.
• PCA solves the eigenvalue equation $S\mathbf{v}_i = \lambda_i\mathbf{v}_i$, where $\mathbf{v}_i$ is a principal direction and $\lambda_i$ is its variance.
• The first principal component is the direction with maximum variance, written as $\mathbf{z}_1 = X_c\mathbf{v}_1$.
• The proportion of variance explained by component $i$ is $\frac{\lambda_i}{\lambda_1 + \lambda_2 + \cdots + \lambda_p}$.
• The cumulative explained variance for the first $k$ components is $\frac{\lambda_1 + \lambda_2 + \cdots + \lambda_k}{\lambda_1 + \lambda_2 + \cdots + \lambda_p}$.
• If variables have very different units or scales, PCA is usually performed on standardized variables using $z = \frac{x - \mu}{\sigma}$.
• A rank-$k$ PCA reconstruction can be written as $\hat{X}_c = Z_k V_k^T$, where $Z_k = X_c V_k$ contains the first $k$ component scores.

Vocabulary

Principal Component

A principal component is a new variable formed as a linear combination of the original variables, chosen to capture as much variance as possible.

Covariance Matrix

A covariance matrix is a square matrix whose entries measure how pairs of variables vary together.

Eigenvector

An eigenvector of a matrix is a nonzero vector whose direction does not change when the matrix acts on it.

Eigenvalue

An eigenvalue is a scalar that tells how much variance is captured in the direction of its corresponding eigenvector in PCA.

Explained Variance

Explained variance is the amount or proportion of total data variation captured by a principal component.

Score

A score is the coordinate of an observation after it has been projected onto a principal component direction.