Anomaly detection is the process of finding observations that do not fit the usual pattern in a data set. These unusual points are often called anomalies or outliers, and they can signal errors, fraud, equipment faults, rare events, or new discoveries. The goal is not just to find strange values, but to decide whether they are meaningful enough to investigate.
This matters because one unusual data point can reveal a major problem or change the conclusion of an analysis.
Statistical methods often compare a value to the center and spread of the data, such as using a z-score or the interquartile range. Distance-based methods look for points that are far from dense clusters or have few nearby neighbors. In real systems, anomaly detection must balance false positives, which waste attention, and false negatives, which miss important events.
Good models also need context because a value that is unusual in one situation may be normal in another.
Key Facts
- An anomaly is an observation that is unusually far from the pattern followed by most of the data.
- Z-score measures how many standard deviations a value is from the mean: z = (x - μ) / σ.
- A common rule flags values with |z| > 3 as potential anomalies in roughly normal data.
- Interquartile range is IQR = Q3 - Q1, and common outlier fences are Q1 - 1.5IQR and Q3 + 1.5IQR.
- Distance-based detection can flag a point if its distance to nearby points is much larger than typical neighbor distances.
- Precision = true positives / predicted positives and recall = true positives / actual positives measure detection performance.
Vocabulary
- Anomaly
- An anomaly is a data point or pattern that differs strongly from what is expected in the data.
- Outlier
- An outlier is an observation that lies far from the main body of a data set.
- Z-score
- A z-score tells how many standard deviations a value is above or below the mean.
- False positive
- A false positive occurs when a normal observation is incorrectly flagged as an anomaly.
- False negative
- A false negative occurs when a real anomaly is missed and labeled as normal.
Common Mistakes to Avoid
- Calling every extreme value an error, which is wrong because some anomalies are valid rare events that contain important information.
- Using z-scores without checking the data shape, which is wrong because z-score rules work best when the data is roughly normal and not heavily skewed.
- Ignoring units and scale before distance calculations, which is wrong because variables with larger numerical ranges can dominate the distance measure.
- Judging a detector only by accuracy, which is wrong because rare anomalies can make a model look accurate even when it misses most important cases.
Practice Questions
- 1 A sensor has mean μ = 50 and standard deviation σ = 4. A new reading is x = 63. Find the z-score and decide whether it is a potential anomaly using |z| > 3.
- 2 For the data set 8, 9, 10, 11, 12, 13, 14, 30, the quartiles are Q1 = 9.5 and Q3 = 13.5. Compute the IQR and the upper outlier fence using Q3 + 1.5IQR, then decide whether 30 is an outlier.
- 3 A credit card transaction is much larger than usual for one customer, but it happens at a store they visit every week and during their normal shopping time. Explain why context matters before labeling it as fraud.