1
$\begingroup$

This is a sister post to the original closed post (here). Since the data transformation part is done after data spliting on the TRAINING data only, I wonder wouldn't such transformation has dependency with how we subsample our data? We can have different transformation results when we pick different portion of training data.

But I personally find it hard to convince myself that: isn't data transformation should be as invariant and generalizable as possible, across different subsamplings of dataset?

Also, as the testing part of the data also represent the real world data. Shouldn't we just transform the data before splitting can we capture more about how the data in 'real world' looks like, and we waste no data? While I accept we only transform the training set and reapply the same on the testing set during prediction in model evaluation/training phrase, isn't it better if during actual deployment we execute transformation on the whole dataset instead and train on all data instead of just sticking to the 'post-splitting transformation' from the model training phrase?

Specifically, for example, if I apply LabelEncoder() from sklearn on the train set, and then I use a new instance of LabelEncoder() on the full dataset, this is legit?

TIA.

$\endgroup$

1 Answer 1

1
$\begingroup$

Yes thats what most data scientists do in the industry. They divide their train & test dataset to find the best model and what works for them. Once they know which model and preprocessing works for them. They apply the same preprocessing and retrain the model with best hyperparams on the whole dataset. So you are thinking in the right direction and thats what is used a lot in the industry.

$\endgroup$
5
  • $\begingroup$ By 'apply the same preprocessing', do you mean for example when they apply normalization on the training dataset, then in final model they apply normalization on the complete dataset? And also for example, if they find categorical encoding works on training dataset, they apply the same categorical encoding on full dataset, despite the '3' in train dataset may refer to category A but to category C in the full dataset setting? $\endgroup$
    – Student
    Apr 12, 2022 at 5:17
  • $\begingroup$ yes apply normalization on train and when it works on whole dataset..If you are retraining your model, this scenario "'3' in train dataset may refer to category A but to category C in the full dataset setting" should be taken care off.. $\endgroup$ Apr 12, 2022 at 5:36
  • $\begingroup$ For example, if I apply LabelEncoder() from sklearn on the train set, and then I use a new instance of LabelEncoder() on the full dataset, this is legit? $\endgroup$
    – Student
    Apr 12, 2022 at 12:05
  • $\begingroup$ Label encoders are tricky, what if some class was missing in train when using train test..but its considered when you train on whole dataset..Now if you change the variables only nee set of hyperparam may be needed $\endgroup$ Apr 12, 2022 at 12:15
  • $\begingroup$ yes, that's the point puzzling me. Updated the question with this specific. $\endgroup$
    – Student
    Apr 12, 2022 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.