The data science workflow is a repeatable process for turning raw data into useful evidence. It matters because good conclusions depend on more than running a model or making a chart. A strong workflow begins with a clear question, uses careful data collection and cleaning, and ends with communication that helps people make decisions.
The process is iterative, so results often lead back to better questions or improved data.
Understanding Statistics: The Data Science Workflow
The first hard part is deciding what evidence would count as an answer. A question about students in one school cannot automatically answer a question about all students in a city. This is the difference between a sample and a population.
The way a sample is chosen matters greatly. A voluntary online poll often attracts people with strong opinions. A survey sent only during school hours can miss students who are absent.
Random selection reduces some bias, but it does not fix every problem. Students should always ask who was included, who was left out, and whether the measurements represent the real situation.
Raw data can look tidy while hiding serious problems. A temperature recorded in degrees Celsius cannot be mixed directly with one recorded in degrees Fahrenheit. A blank cell might mean a person skipped a question, a sensor failed, or the value was truly zero.
Those cases need different treatment. Removing every unusual value can be harmful because an outlier may reveal an important event, such as a faulty machine or an extreme weather day.
Keep a record of each cleaning decision. This record makes the work reproducible and helps others judge whether a result depends on a particular choice.
Patterns in data need careful interpretation. A graph can show that two variables change together, yet it cannot prove that one causes the other. Ice cream sales and sunburn cases may both rise in summer because hot weather affects both.
A hidden factor like weather is called a confounding variable. Comparisons can be misleading when groups differ in important ways. For example, a treatment group may appear healthier simply because it started with younger people.
Good analysis compares like with like when possible. Experiments with random assignment are especially useful for studying cause, though they are not always practical or ethical.
Models are judged by how well they work on cases they did not learn from. If a model is checked using the same data that built it, it may memorize quirks instead of finding a general pattern. This problem is called overfitting.
Holding back test data gives a more honest estimate of performance. Analysts should inspect errors, not just one overall score. A prediction that is usually close may still fail badly for a smaller group of people.
Results should include uncertainty and limits in plain language. A useful conclusion states what the data supports, what remains unknown, and what decision the evidence can reasonably inform.
Key Facts
- A clear statistical question identifies the population, variables, and goal of the analysis.
- Data cleaning includes handling missing values, correcting errors, removing duplicates, and checking units.
- Exploratory data analysis uses summaries and graphs such as mean, median, standard deviation, histograms, and scatterplots.
- Mean = sum of values / number of values.
- A common train-test split is 80 percent training data and 20 percent testing data for evaluating model performance.
- Prediction error for one case can be written as residual = observed value - predicted value.
Vocabulary
- Data Science Workflow
- A structured sequence of steps used to define a problem, prepare data, analyze patterns, build models, and communicate results.
- Exploratory Data Analysis
- The process of using graphs and summary statistics to understand patterns, outliers, and relationships in data.
- Feature
- A variable used as an input to a statistical model or machine learning algorithm.
- Model
- A mathematical or computational representation used to describe data, explain relationships, or make predictions.
- Residual
- The difference between an observed value and the value predicted by a model.
Common Mistakes to Avoid
- Starting with a vague question is wrong because the data, variables, and analysis method cannot be chosen well without a clear goal.
- Skipping data cleaning is wrong because missing values, duplicate records, and inconsistent units can create misleading patterns.
- Using the test data while building the model is wrong because it makes performance look better than it will be on new data.
- Reporting only a final number is wrong because conclusions need context, uncertainty, assumptions, and clear visual communication.
Practice Questions
- 1 A dataset has 1,250 rows. If 80 percent is used for training and 20 percent is used for testing, how many rows go into each set?
- 2 A model predicts house prices of 210,000 dollars, 315,000 dollars, and 420,000 dollars. The observed prices are 200,000 dollars, 330,000 dollars, and 400,000 dollars. Find each residual using residual = observed value - predicted value.
- 3 A team finds that a prediction model works well on old customer data but poorly on new customer data. Explain which parts of the data science workflow should be revisited and why.