Dimensionality reduction is a way to simplify data that has many features, such as pixels in an image, words in a document, or measurements from a sensor. In machine learning, too many features can make patterns harder to see and models slower to train. Reducing dimensions helps turn messy, high-dimensional data into a simpler map while keeping the most important information.
This matters because simpler data is easier to visualize, store, compare, and use for prediction.
A dimensionality reduction method looks for structure in the data, such as directions where values vary the most or groups of points that are close together. It then creates new coordinates that describe the data with fewer numbers, often 2 or 3 instead of hundreds or thousands. PCA is a common linear method that projects data onto directions of maximum variation, while t-SNE and UMAP are nonlinear methods often used to make visual maps of clusters.
The goal is not to keep every detail, but to preserve the patterns that help humans and algorithms understand the data.
Key Facts
- Dimensionality reduction changes data from many features to fewer features while trying to keep important patterns.
- Original data can be written as x = [x1, x2, x3, ..., xn], where n is the number of features.
- A reduced representation can be written as z = [z1, z2, ..., zk], where k < n.
- PCA chooses directions called principal components that capture the largest variance in the data.
- Variance of a feature is Var(x) = sum((xi - mean)^2) / n.
- Dimensionality reduction can improve visualization, reduce noise, speed up training, and help reveal clusters.
Vocabulary
- Dimension
- A dimension is one feature or measurement used to describe a data point.
- Feature
- A feature is an input value, such as height, color, word count, or pixel brightness, used by a machine learning model.
- Projection
- A projection is a way of mapping data from a higher-dimensional space onto a lower-dimensional space.
- Principal Component
- A principal component is a new direction in the data that captures as much variation as possible.
- Cluster
- A cluster is a group of data points that are close together or share similar features.
Common Mistakes to Avoid
- Thinking dimensionality reduction always improves accuracy. It can help, but removing dimensions can also remove useful information needed by a model.
- Confusing dimensions with the number of data points. Dimensions are features, while data points are the examples being measured.
- Assuming a 2D map shows exact distances from the original data. A reduced map is an approximation, so some distances and shapes may be distorted.
- Using dimensionality reduction before scaling features when the method is sensitive to size. Large-number features can dominate the result unless the data is normalized or standardized.
Practice Questions
- 1 A dataset has 1,000 images, and each image has 28 by 28 grayscale pixels. How many dimensions does each image have if each pixel is one feature?
- 2 A machine learning dataset has 120 features per example. After dimensionality reduction, each example has 10 new features. How many features were removed, and what percent reduction is this?
- 3 A class uses dimensionality reduction to make a 2D scatterplot of students' music preferences from 30 survey questions. Explain why nearby points may suggest similar preferences, but why the map should not be treated as a perfect copy of the original 30-dimensional data.