0
$\begingroup$

Should feature scaling/standardization/normalization be done before or after feature selection, and before or after data splitting?

I am confused about the order in which the various pre-processing steps should be done

$\endgroup$

1 Answer 1

2
$\begingroup$

Some feature selection methods will depend on the scale of the data, in which case it seems best to scale beforehand. Other methods won't depend on the scale, in which case it doesn't matter.

All preprocessing should be done after the test split. There are some cases where it won't make a difference, but if you're uncertain it's safer to do everything after splitting. The test set is supposed to act as data your model will see in production; you won't have access to that data to help define scale (or anything else), so don't use it that way while training.

$\endgroup$
9
  • $\begingroup$ But if for example I apply a log transformation then the test set won't be log transformed and would live in another space rendering very low performance. $\endgroup$
    – Caterina
    May 2, 2022 at 15:50
  • $\begingroup$ @Caterina You apply the same transformation to the test set, but using training set statistics. (For log-transform there is no "statistic" to be gleaned from the training data, except maybe the fact that you thought the logarithm would be beneficial.) $\endgroup$
    – Ben Reiniger
    May 2, 2022 at 16:10
  • $\begingroup$ Gotcha, so for example min_max scaler should be applied in each iteration of the cross-validation procedure right? since the min and max won't be the same in each split $\endgroup$
    – Caterina
    May 2, 2022 at 16:13
  • $\begingroup$ Yes; set the "min" and "max" statistic according to the training folds, then use that to transform both the training and test folds. (This is done in sklearn by using fit_transform on training folds and just transform on testing folds, and that's handled by all of the cross-validation methods.) $\endgroup$
    – Ben Reiniger
    May 2, 2022 at 16:16
  • 1
    $\begingroup$ It depends on what method of the pipeline you call (see last comment for how cross-validation methods like cross_val_score or GridSearchCV will call the pipelin). See stackoverflow.com/a/68285130/10495893 for how the pipeline calls individual steps. $\endgroup$
    – Ben Reiniger
    May 2, 2022 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.