Should feature scaling/standardization/normalization be done before or after feature selection, and before or after data splitting?
I am confused about the order in which the various pre-processing steps should be done
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
9
Some feature selection methods will depend on the scale of the data, in which case it seems best to scale beforehand. Other methods won't depend on the scale, in which case it doesn't matter.
All preprocessing should be done after the test split. There are some cases where it won't make a difference, but if you're uncertain it's safer to do everything after splitting. The test set is supposed to act as data your model will see in production; you won't have access to that data to help define scale (or anything else), so don't use it that way while training.
answered May 2, 2022 at 15:30
-
$\begingroup$ But if for example I apply a log transformation then the test set won't be log transformed and would live in another space rendering very low performance. $\endgroup$– CaterinaCommented May 2, 2022 at 15:50
-
$\begingroup$ @Caterina You apply the same transformation to the test set, but using training set statistics. (For log-transform there is no "statistic" to be gleaned from the training data, except maybe the fact that you thought the logarithm would be beneficial.) $\endgroup$– Ben Reiniger ♦Commented May 2, 2022 at 16:10
-
$\begingroup$ Gotcha, so for example min_max scaler should be applied in each iteration of the cross-validation procedure right? since the min and max won't be the same in each split $\endgroup$– CaterinaCommented May 2, 2022 at 16:13
-
$\begingroup$ Yes; set the "min" and "max" statistic according to the training folds, then use that to transform both the training and test folds. (This is done in sklearn by using
fit_transformon training folds and justtransformon testing folds, and that's handled by all of the cross-validation methods.) $\endgroup$– Ben Reiniger ♦Commented May 2, 2022 at 16:16 -
1$\begingroup$ It depends on what method of the pipeline you call (see last comment for how cross-validation methods like
cross_val_scoreorGridSearchCVwill call the pipelin). See stackoverflow.com/a/68285130/10495893 for how the pipeline calls individual steps. $\endgroup$– Ben Reiniger ♦Commented May 2, 2022 at 16:26