High-dimensional data has many variables, such as height, income, test scores, and sensor readings all measured for the same objects. As dimensions increase, data becomes harder to visualize, distances become less intuitive, and patterns can be hidden by noise. This difficulty is often called the curse of dimensionality, and it matters in statistics, machine learning, physics experiments, biology, and image analysis.
Dimensionality reduction helps simplify data while keeping the most important structure.
Key Facts
- A data matrix X often has n rows for observations and p columns for variables.
- PCA usually starts by centering each variable: x_centered = x - mean(x).
- The covariance matrix for centered data is C = (1/(n - 1)) X^T X.
- Principal components are eigenvectors of the covariance matrix: C v = lambda v.
- The variance captured by a principal component equals its eigenvalue lambda.
- Explained variance ratio for component i is lambda_i / (lambda_1 + lambda_2 + ... + lambda_p).
Vocabulary
- Dimensionality
- Dimensionality is the number of variables or features used to describe each data point.
- Curse of dimensionality
- The curse of dimensionality is the set of problems that occur when data becomes sparse and harder to analyze as the number of dimensions grows.
- Principal component
- A principal component is a new axis formed from the original variables that points in a direction of high variance.
- Eigenvalue
- An eigenvalue in PCA measures how much variance is captured along its matching eigenvector.
- Projection
- Projection is the process of mapping data points onto a lower-dimensional line, plane, or space.
Common Mistakes to Avoid
- Skipping centering before PCA is wrong because the first component may point toward the mean offset instead of the main direction of variation.
- Assuming PCA always improves prediction is wrong because PCA keeps directions with large variance, not necessarily directions most related to the target variable.
- Comparing PCA results from variables with very different units without scaling can be wrong because variables with larger numerical ranges can dominate the components.
- Thinking the first two principal components always preserve all important information is wrong because the lost components may still contain meaningful structure or rare signals.
Practice Questions
- 1 A centered data set has covariance matrix eigenvalues 9, 4, 1, and 1. What percent of the total variance is captured by the first two principal components?
- 2 A data table has 200 observations and 50 features. After PCA, you keep 5 principal components. By what factor has the feature dimension been reduced?
- 3 A data set has one variable measured in dollars from 0 to 100000 and another measured in meters from 0 to 2. Explain why standardizing the variables before PCA may be important.