A support vector machine, or SVM, is a machine learning method that sorts data into groups by drawing the best possible boundary between them. It is useful for classification tasks such as detecting spam, recognizing simple images, or separating medical test results into categories. The main idea is not just to separate the data, but to separate it with the widest safe space between the groups.
This makes the model more likely to work well on new data it has not seen before.
In a 2D picture, an SVM draws a line called a decision boundary between two classes of points. The closest points to that line are called support vectors, and they control where the boundary goes. The distance from the boundary to these closest points is called the margin, and the SVM tries to make this margin as large as possible.
When the data cannot be separated by a straight line, SVMs can use kernels to create curved boundaries in a transformed feature space.
Key Facts
- An SVM classifies data by finding a decision boundary that separates classes.
- For a line in 2D, a decision boundary can be written as w1x1 + w2x2 + b = 0.
- The predicted class is often based on the sign of f(x) = w · x + b.
- The margin is the distance between the decision boundary and the nearest training points.
- Support vectors are the closest data points to the boundary and have the biggest influence on the model.
- For linearly separable data, the margin width is 2 / ||w||, so maximizing the margin means minimizing ||w||.
Vocabulary
- Support Vector Machine
- A machine learning algorithm that finds a boundary with the widest margin to separate data into classes.
- Decision Boundary
- The line, plane, or surface that a model uses to separate one predicted class from another.
- Support Vector
- A training data point closest to the decision boundary that helps determine the boundary's position.
- Margin
- The gap between the decision boundary and the nearest data points from each class.
- Kernel
- A function that lets an SVM create a nonlinear boundary by comparing data as if it were in a higher-dimensional space.
Common Mistakes to Avoid
- Choosing the line that merely separates the points, not the one with the widest margin, is wrong because SVMs look for the most confident separation, not just any separation.
- Thinking every data point controls the boundary is wrong because only the support vectors have the strongest effect on where the boundary is placed.
- Assuming SVMs can only draw straight lines is wrong because kernels allow SVMs to make curved decision boundaries for nonlinear patterns.
- Ignoring feature scaling is wrong because SVMs use distances, so features with large numerical ranges can unfairly dominate the model.
Practice Questions
- 1 A decision boundary is f(x) = 2x1 - x2 + 1. For the point (3, 4), calculate f(x) and predict the class if positive values mean Class A and negative values mean Class B.
- 2 An SVM has ||w|| = 0.5 for a linearly separable dataset. Use margin width = 2 / ||w|| to find the margin width.
- 3 Two possible separating lines both classify all training points correctly. Line 1 has a small margin, and Line 2 has a large margin. Explain which line an SVM would choose and why.