12
$\begingroup$

I am working on a multi-class classification problem, with ~65 features and ~150K instances. 30% of features are categorical and the rest are numerical (continuous). I understand that standardization or normalization should be done after splitting the data into train and test subsets, but I am not still sure about the imputation process. For the classification task, I am planning to use Random Forest, Logistic Regression, and XGBOOST (which are not distance-based).

Could someone please explain which should come first? Split > imputation or imputation>split? In case that split>imputation is correct, should I follow imputation>standardization or standardization>imputation?

Improve this question
$\endgroup$
2
  • $\begingroup$ Imputation --> standardization or standardization --> imputation will depend on what method of imputation you use, in particular, if the imputation method is sensitive to the scale/magnitude of your predictors (like kNN imputation, for example). If you were to use simple mean imputation then it probably makes more sense to impute first, and then standardize. If you use something like kNN imputation then it is necessary to standardize and then impute due to how distance calculations work... $\endgroup$
    – aranglol
    Jun 4, 2019 at 1:15
  • $\begingroup$ There really isn't a clear order (in my opinion) unless you give specifics as to what imputation method you intend to use. $\endgroup$
    – aranglol
    Jun 4, 2019 at 1:15

2 Answers 2

Reset to default
18
$\begingroup$

Always split before you do any data pre-processing. Performing pre-processing before splitting will mean that information from your test set will be present during training, causing a data leak.

Think of it like this, the test set is supposed to be a way of estimating performance on totally unseen data. If it affects the training, then it will be partially seen data.

I don't think the order of scaling/imputing is as strict. I would impute first if the method might throw of the scaling/centering.

Your steps should be:

  1. Splitting
  2. Imputing
  3. Scaling

Here are some related questions to support this:

Imputation before or after splitting into train and test?

Imputation of missing data before or after centering and scaling?

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ Thank you for adding those references, they were very helpful. I am persuaded, and have removed my answer. $\endgroup$
    – Upper_Case
    Jun 3, 2019 at 16:39
  • $\begingroup$ Glad it helped, @Upper_Case. I find it odd that ISLR had examples where this was not the case. $\endgroup$
    – Simon Larsson
    Jun 3, 2019 at 16:46
  • $\begingroup$ The copy I have is a first-printing, so possibly it was updated later, and the example I referenced doesn't deal with imputation, so details may differ with that element. I'm also not clear on how "bad" it is to do it one way versus the other (I agree about the test-training "leakage", which is bad, but post-split data transformation causes arbitrary data segmentation features to "leak" into the model, which is also bad). As I'm not sure which is worse, especially in the general case, I'm deferring to the votes from CrossValidated.SE. $\endgroup$
    – Upper_Case
    Jun 3, 2019 at 16:51
  • $\begingroup$ @Upper_Case Can you elaborate what "arbitrary data segmentation" features means? Like the training set having a mean/standard deviation that is not reflective of the entire population at whole? I am curious to understand what you mean. $\endgroup$
    – aranglol
    Jun 4, 2019 at 1:18
  • $\begingroup$ @aranglol Your intuition about my meaning is correct: dividing the data into test and training sets randomly doesn't necessarily reproduce the distribution of the original data. Using data which is derived from that arbitrary assignment (such as imputed values after splitting) then incorporates information not actually present in the underlying process being modeled. And the magnitude of the issue depends on observed variance in the variable and which observations happen to be assigned to each split. $\endgroup$
    – Upper_Case
    Jun 4, 2019 at 14:17
3
$\begingroup$

If you impute/standardize before splitting and then split into train/test you are leaking data from your test set (that is supposed to be completely withheld) into your training set. This will yield extremely biased results on model performance.

The correct way is to split your data first, and to then use imputation/standardization (the order will depend on if the imputation method requires standardization).

The key here is that you are learning everything from the training set and then "predicting" on to the test set. For nornalization/standardization, you learn the sample mean and sample standard deviation from the training set, treat them as constants, and using these learned values you transform the test set. You don't use the test set mean or the test standard deviation in any of these calculations.

For imputation the idea is similar. You learn the required parameters from the training set only and then predict the required test set values.

This way your performance metrics will not be biased optimistically by your methods inadverdently seeing the test set observations.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.