2
$\begingroup$

Say you have a binary classification problem, and a dataset with 20,000 observations and 20 columns. The target variable is very imbalanced, there are are missing values, skewed distributions, outliers, etc.

My question is, in a general sense, what order should these data preprocessing steps be performed?

Fill in missing values, Normalize/standardize data, Deal with skewness, Deal with outliers, Balance target variable classes

$\endgroup$
1
$\begingroup$

You're asking a complex question that is dependant on what you aim to find.

If there are missing values, are they in columns relevant to the result you're looking for? If not then they needn't be filled at all, so the order does not matter. That being said, if a column with missing values is pertinent to your desired result, then it's often prudent to fill the missing dataset before analysing it further.

Again, outliers need to be addressed in relation to what you're trying to ascertain. In many cases, presence of the outliers can highlight a result in itself. Which may not answer what you're asking, but instead make you question the accuracy of the data itself, or make you determine to include them or exclude them.

You need to be more specific in what it is you're trying to ascertain. Then approach each of the preprocessing steps in the order that is relevant to you. This changes in many different situations, so it's difficult to give a definitive answer.

The tutorial at kaggle to resolve the Titanic Data Science Solutions lays out a good methodology for approaching such problem, and reinforces the concept of sometimes reorganising workflow tasks.

Workflow stages

The competition solution workflow goes through seven stages described in the Data Science Solutions book.

  • Question or problem definition.
  • Acquire training and testing data.
  • Wrangle, prepare, cleanse the data.
  • Analyze, identify patterns, and explore the data.
  • Model, predict and solve the problem.
  • Visualize, report, and present the problem solving steps and final solution.
  • Supply or submit the results.

The workflow indicates general sequence of how each stage may follow the other. However there are use cases with exceptions.

  • We may combine mulitple workflow stages. We may analyze by visualizing data.
  • Perform a stage earlier than indicated. We may analyze data before and after wrangling.
  • Perform a stage multiple times in our workflow. Visualize stage may be used multiple times.
  • Drop a stage altogether. We may not need supply stage to productize or service enable our dataset for a competition.

Kaggle - Titanic Data Science Solutions

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.