Real data rarely arrives ready for statistics. It often contains missing values, duplicate records, outliers, spelling variations, inconsistent units, and dates written in different formats. Data cleaning and wrangling matter because even a powerful model or test can give misleading results when the input data is messy.
In many real projects, preparing the data takes more time than running the final analysis.
Understanding Statistics: Data Cleaning and Wrangling
Cleaning starts with understanding what one row is meant to represent. A row might be one student, one shop sale, one patient visit, or one reading from a weather sensor. If that meaning changes halfway through a table, counts and averages lose their meaning.
A useful first step is to inspect column names, data types, ranges, and a small sample of rows. Numbers stored as text can block calculations.
A date stored as several different text styles can sort in the wrong order. A category such as "yes", "Yes", and "Y" may be treated as three separate groups unless it is standardized.
Missing data needs thought before it is filled or removed. A blank can mean that a measurement was not taken, a person chose not to answer, or the value does not apply. These cases can lead to different decisions.
For example, a missing income response may be related to income itself, so replacing every blank with an average can distort the result. Removing rows can be just as risky if the removed records belong mostly to one group.
Keep a separate label for missing values when it carries information. Record how many values are missing in each column and check whether missingness is concentrated in a certain time, place, or group.
Outliers are not automatically mistakes. A very high electricity bill could come from a typing error, a factory building, or a real heat wave. Statistical checks can flag unusual values by comparing them with the center and spread of the data.
The interquartile range is helpful because it is less affected by extreme values than the mean. A z score is useful when the data is roughly bell shaped. After a value is flagged, check its source before changing it.
Look for impossible values such as a negative age, unit mixups such as grams entered where kilograms were expected, and misplaced decimal points. If a correction is made, preserve the original value in a raw copy.
Wrangling reshapes data so it can answer a specific question. A school may receive monthly attendance in twelve separate columns, but a graph over time works better when each month becomes a row with one attendance value. Tables can be joined using an ID, such as a student number or product code.
This requires care because repeated IDs can create extra rows and inflate totals. Keep a clear record of every step, including filters, recoded categories, removed duplicates, and changed units. In class projects, compare summaries before and after cleaning.
Large changes in the mean, median, group sizes, or graph shape are signals to investigate. Good preparation does not make data perfect. It makes the limits of the data visible and helps later conclusions stay honest.
Key Facts
- Percent missing = number of missing values / total number of values x 100%
- Duplicate rate = number of duplicate records / total number of records x 100%
- Mean imputation replaces a missing value with x̄ = sum of values / n
- Z-score for outlier checking: z = (x - μ) / σ
- Interquartile range rule: outliers are often values below Q1 - 1.5IQR or above Q3 + 1.5IQR
- Tidy data principle: each variable is a column, each observation is a row, and each value is a single cell
Vocabulary
- Data cleaning
- Data cleaning is the process of finding and fixing errors, missing values, duplicates, and inconsistencies in a dataset.
- Data wrangling
- Data wrangling is the process of reshaping, combining, filtering, and organizing data so it is ready for analysis.
- Missing value
- A missing value is a blank, null, or unavailable entry where a measurement or category should appear.
- Outlier
- An outlier is a data value that is unusually far from the rest of the dataset and may be an error or a meaningful extreme case.
- Tidy data
- Tidy data is a structured format where variables are columns, observations are rows, and each cell contains one value.
Common Mistakes to Avoid
- Deleting all rows with missing values, because this can shrink the sample and create bias if the missingness is not random.
- Treating every outlier as an error, because some outliers are valid observations that reveal important patterns or rare events.
- Mixing units or formats in one column, because calculations become invalid when values like meters and centimeters or MM/DD/YYYY and DD/MM/YYYY are combined without conversion.
- Cleaning data without keeping a record of changes, because the analysis becomes hard to reproduce and mistakes cannot be traced later.
Practice Questions
- 1 A dataset has 500 rows and 35 rows contain at least one missing value. What percent of rows contain missing values?
- 2 A file has 1,200 customer records. After checking unique customer IDs, you find 48 duplicate records. What is the duplicate rate as a percent?
- 3 A temperature dataset contains one value of 200 degrees Celsius for an ordinary classroom sensor. Explain two checks you would make before deciding whether to remove, correct, or keep this value.