K-Nearest Neighbors, often called KNN, is a simple machine learning method used to classify new data based on examples it has already seen. It is useful because it connects geometry, statistics, and computer science in a visual way. If a new point is placed on a graph, KNN looks at the closest known points and lets them vote on the new point's label.
This makes it a good first model for students learning how computers can make predictions from data.
KNN works by measuring distance between data points, often using the distance formula on a coordinate grid. The value of k tells the model how many nearby points to consider before making a decision. A small k can react strongly to one unusual point, while a larger k can give a smoother and more stable prediction.
KNN can be used for tasks like identifying handwritten digits, recommending items, sorting images, or predicting categories from measured features.
Key Facts
- KNN stands for K-Nearest Neighbors.
- The model predicts a label by using the labels of the k closest training points.
- Distance in 2D is often measured with d = sqrt((x2 - x1)^2 + (y2 - y1)^2).
- For classification, the predicted class is usually the majority vote among the nearest neighbors.
- For regression, the predicted value can be the average of the nearest neighbors: prediction = sum(values) / k.
- Choosing k matters: small k can be noisy, while large k can hide local patterns.
Vocabulary
- K-Nearest Neighbors
- A machine learning algorithm that predicts a new data point by comparing it with the closest known data points.
- Training Data
- Examples with known inputs and labels that a machine learning model uses to make future predictions.
- Feature
- A measurable property of a data point, such as height, color value, speed, or x-coordinate.
- Classification
- A prediction task where the model assigns a data point to a category or class.
- Distance Metric
- A rule for calculating how far apart two data points are in feature space.
Common Mistakes to Avoid
- Choosing k without testing it is a mistake because different data sets need different neighborhood sizes for accurate predictions.
- Forgetting to scale features is a mistake because a feature with large numbers can dominate the distance calculation even if it is not more important.
- Using an even k for two-class classification can be a mistake because it can create ties in the nearest-neighbor vote.
- Thinking KNN learns a formula during training is a mistake because KNN stores the training examples and compares new points to them when making a prediction.
Practice Questions
- 1 A new point is at (4, 3). A known red point is at (1, 3), and a known blue point is at (4, 7). Use the distance formula to find which known point is closer.
- 2 A KNN model uses k = 5. The five nearest neighbors have labels cat, dog, cat, bird, cat. What class does the model predict, and why?
- 3 A data set uses height in centimeters and age in years as features. Explain why feature scaling might be important before using KNN.