Real data rarely arrives ready for statistics. It often contains missing values, duplicate records, outliers, spelling variations, inconsistent units, and dates written in different formats. Data cleaning and wrangling matter because even a powerful model or test can give misleading results when the input data is messy.
In many real projects, preparing the data takes more time than running the final analysis.
Key Facts
- Percent missing = number of missing values / total number of values x 100%
- Duplicate rate = number of duplicate records / total number of records x 100%
- Mean imputation replaces a missing value with x̄ = sum of values / n
- Z-score for outlier checking: z = (x - μ) / σ
- Interquartile range rule: outliers are often values below Q1 - 1.5IQR or above Q3 + 1.5IQR
- Tidy data principle: each variable is a column, each observation is a row, and each value is a single cell
Vocabulary
- Data cleaning
- Data cleaning is the process of finding and fixing errors, missing values, duplicates, and inconsistencies in a dataset.
- Data wrangling
- Data wrangling is the process of reshaping, combining, filtering, and organizing data so it is ready for analysis.
- Missing value
- A missing value is a blank, null, or unavailable entry where a measurement or category should appear.
- Outlier
- An outlier is a data value that is unusually far from the rest of the dataset and may be an error or a meaningful extreme case.
- Tidy data
- Tidy data is a structured format where variables are columns, observations are rows, and each cell contains one value.
Common Mistakes to Avoid
- Deleting all rows with missing values, because this can shrink the sample and create bias if the missingness is not random.
- Treating every outlier as an error, because some outliers are valid observations that reveal important patterns or rare events.
- Mixing units or formats in one column, because calculations become invalid when values like meters and centimeters or MM/DD/YYYY and DD/MM/YYYY are combined without conversion.
- Cleaning data without keeping a record of changes, because the analysis becomes hard to reproduce and mistakes cannot be traced later.
Practice Questions
- 1 A dataset has 500 rows and 35 rows contain at least one missing value. What percent of rows contain missing values?
- 2 A file has 1,200 customer records. After checking unique customer IDs, you find 48 duplicate records. What is the duplicate rate as a percent?
- 3 A temperature dataset contains one value of 200 degrees Celsius for an ordinary classroom sensor. Explain two checks you would make before deciding whether to remove, correct, or keep this value.