Feature scaling changes the numerical range of variables so they can be compared fairly by a statistical model. It matters because many algorithms treat larger numbers as more important, even when the larger scale is just a measurement choice. A height measured in centimeters and an income measured in dollars can have very different magnitudes, but both may carry useful information.
Scaling helps the model focus on patterns rather than units.
Key Facts
- Standardization: z = (x - mean) / standard deviation
- Min-max normalization: x' = (x - min) / (max - min)
- After standardization, a feature has mean 0 and standard deviation 1, unless the data distribution changes later.
- After min-max normalization, values usually fall between 0 and 1 for the data used to compute the minimum and maximum.
- Distance-based methods such as k-nearest neighbors and k-means are sensitive to feature scale because distance = sqrt((x2 - x1)^2 + (y2 - y1)^2).
- Fit scaling parameters on the training set only, then apply the same mean, standard deviation, minimum, or maximum to validation and test data.
Vocabulary
- Feature scaling
- Feature scaling is the process of transforming variables so their numerical sizes are more comparable.
- Standardization
- Standardization converts a value into a z-score by subtracting the mean and dividing by the standard deviation.
- Min-max normalization
- Min-max normalization rescales values using the smallest and largest values so they usually lie between 0 and 1.
- Distance-based algorithm
- A distance-based algorithm makes decisions using distances between data points, so features with larger scales can dominate the result.
- Data leakage
- Data leakage happens when information from validation or test data is used during training, giving an overly optimistic result.
Common Mistakes to Avoid
- Scaling the entire dataset before splitting into training and test sets is wrong because it lets test data influence the transformation. Compute scaling values from the training set only.
- Using min-max normalization when extreme outliers are present can be wrong because one unusually large or small value can squeeze most data into a tiny range. Consider standardization or robust scaling instead.
- Forgetting to scale new prediction data is wrong because the model expects inputs on the same scale used during training. Apply the saved training-set transformation to every new example.
- Scaling features that should stay as labels or categories is wrong because target variables and category codes may lose their meaning. Scale only appropriate numerical input features unless there is a specific reason.
Practice Questions
- 1 A feature has values 10, 14, 18, 22, 26. Use min-max normalization to scale the value 18.
- 2 A feature has mean 50 and standard deviation 8. Find the standardized z-score for x = 62.
- 3 A dataset has two features: age ranges from 18 to 80, and annual income ranges from 20,000 to 200,000. Explain why k-nearest neighbors may perform poorly without scaling, and name one scaling method that could help.