Principal Component Analysis, or PCA, is a machine learning and statistics method for simplifying complicated data. It helps find the most important patterns in a data set with many measurements, such as height, weight, age, and test scores. PCA is useful because computers and people can understand data more easily when the main information is shown with fewer dimensions.
It is often used before visualization, compression, or other machine learning steps.
PCA works by rotating the coordinate system so that the first new axis points in the direction where the data varies the most. The next axis points in the direction of the next largest variation, while staying perpendicular to the first axis. These new axes are called principal components, and each one captures some amount of the total variation in the data.
By keeping only the first few components, PCA can reduce noise and make hidden patterns easier to see.
Key Facts
- PCA finds new axes called principal components that point in the directions of greatest data variation.
- The first principal component captures the largest possible variance in the data.
- The second principal component captures the next largest variance and is perpendicular to the first.
- Variance measures spread: variance = sum((x - mean)^2) / n.
- A data point can be projected onto a component using projection = x dot unit vector.
- Dimensionality reduction keeps the most important components and drops components with small variance.
Vocabulary
- Principal Component
- A new axis found by PCA that shows an important direction of variation in the data.
- Variance
- Variance is a measure of how spread out data values are from their mean.
- Projection
- Projection is the process of placing a point onto a line or plane to show its position in a simpler space.
- Dimensionality Reduction
- Dimensionality reduction is the process of representing data with fewer features while keeping important information.
- Feature
- A feature is one measured property of each data item, such as temperature, height, or pixel brightness.
Common Mistakes to Avoid
- Forgetting to center the data first is wrong because PCA depends on variation around the mean, not variation around the original coordinate axes.
- Thinking PCA chooses the most important original features is wrong because PCA creates new combined directions from the original features.
- Keeping too many principal components can be a mistake because it may keep noise and make the data less simple to interpret.
- Assuming PCA always improves machine learning results is wrong because PCA can remove information that a model needs for a specific prediction task.
Practice Questions
- 1 A data set has total variance 100. The first principal component explains 65 units of variance and the second explains 20 units. What percent of the total variance is explained by the first two components together?
- 2 A centered 2D data point is x = (6, 8). A unit principal component points in the direction u = (0.6, 0.8). Use projection = x dot u to find the point's coordinate along this principal component.
- 3 A data set has 10 features, but the first two principal components explain 92 percent of the variance. Explain why a scientist might choose to plot only those two components and what information might be lost.