Principal component analysis, or PCA, is a method for reducing many related variables into a smaller number of new variables called principal components. It helps students summarize high-dimensional data, identify major patterns, and visualize data more easily. This cheat sheet connects the setup, matrix calculations, and interpretation steps needed to use PCA correctly.
It is useful when data has many measurements that may overlap or be strongly correlated.
The core idea is to center or standardize the data, compute a covariance or correlation matrix, and find its eigenvalues and eigenvectors. Eigenvectors give the directions of the principal components, while eigenvalues measure how much variance each component explains. Scores are found by projecting the data onto the selected component directions.
The proportion of explained variance helps decide how many components to keep.
Key Facts
- For a data matrix with observations and variables, the centered data matrix is , where each column mean is subtracted.
- The sample covariance matrix for centered data is .
- PCA solves the eigenvalue equation , where is a principal direction and is its variance.
- The first principal component is the direction with maximum variance, written as .
- The proportion of variance explained by component is .
- The cumulative explained variance for the first components is .
- If variables have very different units or scales, PCA is usually performed on standardized variables using .
- A rank- PCA reconstruction can be written as , where contains the first component scores.
Vocabulary
- Principal Component
- A principal component is a new variable formed as a linear combination of the original variables, chosen to capture as much variance as possible.
- Covariance Matrix
- A covariance matrix is a square matrix whose entries measure how pairs of variables vary together.
- Eigenvector
- An eigenvector of a matrix is a nonzero vector whose direction does not change when the matrix acts on it.
- Eigenvalue
- An eigenvalue is a scalar that tells how much variance is captured in the direction of its corresponding eigenvector in PCA.
- Explained Variance
- Explained variance is the amount or proportion of total data variation captured by a principal component.
- Score
- A score is the coordinate of an observation after it has been projected onto a principal component direction.
Common Mistakes to Avoid
- Forgetting to center the data is wrong because PCA directions depend on variation around the mean, not the raw location of the data.
- Using covariance PCA when variables have very different units can be wrong because large-scale variables may dominate the components even if they are not more important.
- Treating principal components as original variables is wrong because each component is a weighted combination of all original variables.
- Choosing components only because their eigenvalues are largest can be incomplete because the decision should also consider cumulative explained variance and the purpose of the analysis.
- Ignoring the signs of eigenvectors incorrectly can cause confusion because the direction may flip without changing the PCA solution, but score interpretations must stay consistent.
Practice Questions
- 1 A centered data matrix has eigenvalues , , and . What proportion of variance is explained by the first principal component?
- 2 For eigenvalues , , , and , how many components are needed to explain at least of the total variance?
- 3 A data point has centered values and the first principal direction is . Find its first principal component score .
- 4 Explain why standardizing variables before PCA may be important when one variable is measured in dollars and another is measured in centimeters.
Understanding Principal Component Analysis Reference
Each observation can be pictured as a point in a space with one axis for each measured feature. With only two features, this is a flat scatter plot. With three features, it is a 3D cloud.
More features create spaces that cannot be drawn, but the same geometry still applies. PCA rotates the viewpoint to line up with the longest patterns in that cloud. The first new axis follows the strongest overall spread.
Later axes describe leftover spread in directions at right angles to earlier ones. This right angle rule prevents two components from describing the same pattern twice.
Preparation strongly affects the result. Centering puts the average observation at the origin, so the analysis measures variation around what is typical rather than variation caused by a nonzero average. Scaling matters when one measurement has much larger numbers than another.
For example, height measured in centimeters may vary by tens, while a test score varies by only a few points. Without standardization, height can dominate simply because of its unit size. Standardizing gives each variable a comparable starting scale.
It does not make every variable equally important in the real world. It only stops units from deciding the answer.
The covariance matrix records how pairs of variables change together. A positive covariance means high values of one variable often occur with high values of another. A negative covariance means one tends to fall when the other rises.
PCA uses all of these relationships at once. A component direction has weights for the original variables. Large positive or negative weights show which variables contribute strongly to that component.
The sign itself is not important. Reversing every sign gives the same direction and the same information. Students should focus on the relative size and pattern of the weights rather than treating a positive sign as automatically good.
Choosing how many components to keep is a judgment about information loss. A scree plot places component number against explained variance. It often drops steeply at first, then levels off.
The bend can suggest that later components mostly contain small details or noise. A chosen set of components can rebuild an approximation of the original data. Differences between the original values and rebuilt values are reconstruction errors.
Small errors mean the reduced representation retains much of the data structure. Important patterns can still be lost, especially patterns that have low variance. PCA favors large spread, not necessarily the feature most useful for prediction or decision making.
Students meet these ideas in image compression, survey analysis, genetics, sports statistics, and sensor data. A photo contains many pixel values, but nearby pixels are often related, so a smaller set of patterns can store much of its appearance. In a school survey, several questions about study habits may move together and form one broad pattern.
PCA is less reliable when relationships are strongly curved, when outliers are extreme, or when categories are mixed with numerical measurements without careful coding. When learning PCA, track the units, inspect unusual observations, interpret component weights in context, and remember that a clear plot can simplify data without proving cause and effect.