Principal Component Analysis, or PCA, is a machine learning and statistics method for simplifying complicated data. It helps find the most important patterns in a data set with many measurements, such as height, weight, age, and test scores. PCA is useful because computers and people can understand data more easily when the main information is shown with fewer dimensions.
It is often used before visualization, compression, or other machine learning steps.
PCA works by rotating the coordinate system so that the first new axis points in the direction where the data varies the most. The next axis points in the direction of the next largest variation, while staying perpendicular to the first axis. These new axes are called principal components, and each one captures some amount of the total variation in the data.
By keeping only the first few components, PCA can reduce noise and make hidden patterns easier to see.
Understanding AI & Machine Learning: Principal Component Analysis Explained
Before PCA is calculated, the data usually needs preparation. Each measurement should describe the same set of cases, such as the same students or products. Missing values need a sensible treatment because PCA cannot interpret a blank measurement by itself.
Most importantly, features are often centered by subtracting their average value. This makes PCA study differences from the typical case rather than the raw size of the numbers. Features may need scaling too.
Imagine a data set with distance in kilometres and mass in grams. The gram values are much larger, so they can dominate the result even if distance is equally important. Standardizing gives each feature a similar starting scale.
The calculation behind PCA examines how measurements change together. If two features tend to rise and fall together, they contain overlapping information. A covariance table records these shared changes for every pair of features.
PCA uses this table to find special directions through the data cloud. In more advanced courses, these directions come from eigenvectors. The matching eigenvalues tell how much spread lies along each direction.
Students do not need to calculate eigenvectors by hand for most projects, but the idea matters. PCA is not choosing the most important original column. It is building new features from weighted mixtures of the original columns.
After choosing a small number of components, each original data point receives new coordinates. This is called its score in the component space. A spreadsheet with hundreds of measurements can then be represented by two or three scores per row.
Plotting the first two scores can reveal clusters, unusual cases, or gradual trends. For example, many exam results across subjects may be summarized by a broad academic performance pattern plus a second pattern that separates science strengths from language strengths. PCA does not assign labels to these groups.
It only shows structure that may be present. A later model or a human investigator must decide what the structure means.
Keeping fewer components always loses some information. Analysts choose a number by checking the explained variance ratio, which states the share of total spread retained. A common goal is to keep enough components to preserve most of the useful variation, though there is no single correct percentage.
The dropped components might mostly contain measurement error, but they can sometimes contain an important small signal. PCA works best when the main relationships are roughly linear. Curved shapes, strong outliers, and categories coded as numbers can give misleading results.
For machine learning, PCA must be fitted using training data only. The same centering, scaling, and component directions are then applied to validation or test data. This avoids accidentally learning information from the test set.
Key Facts
- PCA finds new axes called principal components that point in the directions of greatest data variation.
- The first principal component captures the largest possible variance in the data.
- The second principal component captures the next largest variance and is perpendicular to the first.
- Variance measures spread: variance = sum((x - mean)^2) / n.
- A data point can be projected onto a component using projection = x dot unit vector.
- Dimensionality reduction keeps the most important components and drops components with small variance.
Vocabulary
- Principal Component
- A new axis found by PCA that shows an important direction of variation in the data.
- Variance
- Variance is a measure of how spread out data values are from their mean.
- Projection
- Projection is the process of placing a point onto a line or plane to show its position in a simpler space.
- Dimensionality Reduction
- Dimensionality reduction is the process of representing data with fewer features while keeping important information.
- Feature
- A feature is one measured property of each data item, such as temperature, height, or pixel brightness.
Common Mistakes to Avoid
- Forgetting to center the data first is wrong because PCA depends on variation around the mean, not variation around the original coordinate axes.
- Thinking PCA chooses the most important original features is wrong because PCA creates new combined directions from the original features.
- Keeping too many principal components can be a mistake because it may keep noise and make the data less simple to interpret.
- Assuming PCA always improves machine learning results is wrong because PCA can remove information that a model needs for a specific prediction task.
Practice Questions
- 1 A data set has total variance 100. The first principal component explains 65 units of variance and the second explains 20 units. What percent of the total variance is explained by the first two components together?
- 2 A centered 2D data point is x = (6, 8). A unit principal component points in the direction u = (0.6, 0.8). Use projection = x dot u to find the point's coordinate along this principal component.
- 3 A data set has 10 features, but the first two principal components explain 92 percent of the variance. Explain why a scientist might choose to plot only those two components and what information might be lost.