High-dimensional data has many variables, such as height, income, test scores, and sensor readings all measured for the same objects. As dimensions increase, data becomes harder to visualize, distances become less intuitive, and patterns can be hidden by noise. This difficulty is often called the curse of dimensionality, and it matters in statistics, machine learning, physics experiments, biology, and image analysis.
Dimensionality reduction helps simplify data while keeping the most important structure.
Understanding Statistics: Dimensionality and PCA
PCA works by changing the directions used to describe the data. The original variables act like axes on a graph. PCA rotates those axes to find new directions that match the main spread of the observations.
The first principal component points along the direction with the greatest variation. The second points along the greatest remaining variation, while staying perpendicular to the first.
Each later component follows the same rule. These new directions are combinations of the original variables, so one component may reflect a pattern involving several measurements at once.
Before PCA, centering is essential because PCA measures variation around an average. Without centering, a variable with a large average can pull the result toward that average rather than toward meaningful differences between observations. Scaling is often important too.
Consider a data set containing mass in kilograms and length in millimetres. Length values may be numerically much larger simply because of their unit. PCA may then focus mostly on length.
Standardising each variable to have a spread of about one gives variables a more equal starting point. Whether to scale depends on the meaning of the units and the scientific goal.
After finding the component directions, each observation can be projected onto them. The resulting numbers are called component scores. A scatter plot of the first two scores can reveal clusters, trends, or unusual observations that were difficult to see in the original measurements.
The numbers used to build each component are called loadings. Loadings help interpret what a component represents.
If several test scores have large positive loadings on the first component, that component may describe overall academic performance. If one group of measurements has positive loadings while another has negative loadings, the component may describe a trade off between them.
Choosing how many components to keep requires judgement. A common approach is to add the fractions of variance explained by the leading components and retain enough to preserve a useful amount of the total variation. A sharp drop in explained variance can suggest a sensible cutoff.
Still, variance is not the same as importance. A small-variance component can contain information needed to separate rare classes or identify a fault in a machine. PCA is also sensitive to outliers, since a few extreme points can strongly change the direction of greatest spread.
Students should check units, missing values, outliers, and the meaning of loadings before trusting a compact PCA picture. PCA helps with compression and exploration, but it does not prove that one variable causes another.
Key Facts
- A data matrix X often has n rows for observations and p columns for variables.
- PCA usually starts by centering each variable: x_centered = x - mean(x).
- The covariance matrix for centered data is C = (1/(n - 1)) X^T X.
- Principal components are eigenvectors of the covariance matrix: C v = lambda v.
- The variance captured by a principal component equals its eigenvalue lambda.
- Explained variance ratio for component i is lambda_i / (lambda_1 + lambda_2 + ... + lambda_p).
Vocabulary
- Dimensionality
- Dimensionality is the number of variables or features used to describe each data point.
- Curse of dimensionality
- The curse of dimensionality is the set of problems that occur when data becomes sparse and harder to analyze as the number of dimensions grows.
- Principal component
- A principal component is a new axis formed from the original variables that points in a direction of high variance.
- Eigenvalue
- An eigenvalue in PCA measures how much variance is captured along its matching eigenvector.
- Projection
- Projection is the process of mapping data points onto a lower-dimensional line, plane, or space.
Common Mistakes to Avoid
- Skipping centering before PCA is wrong because the first component may point toward the mean offset instead of the main direction of variation.
- Assuming PCA always improves prediction is wrong because PCA keeps directions with large variance, not necessarily directions most related to the target variable.
- Comparing PCA results from variables with very different units without scaling can be wrong because variables with larger numerical ranges can dominate the components.
- Thinking the first two principal components always preserve all important information is wrong because the lost components may still contain meaningful structure or rare signals.
Practice Questions
- 1 A centered data set has covariance matrix eigenvalues 9, 4, 1, and 1. What percent of the total variance is captured by the first two principal components?
- 2 A data table has 200 observations and 50 features. After PCA, you keep 5 principal components. By what factor has the feature dimension been reduced?
- 3 A data set has one variable measured in dollars from 0 to 100000 and another measured in meters from 0 to 2. Explain why standardizing the variables before PCA may be important.