Recently I've been thinking about the proper use of encoding within cross-validation scheme. The customarily advised way of encoding features is:
- Split the data into train and test (hold-out) set
- Fit the encoder (either
OneHotEncoder) on the train set
- Transform both the train and test set using fitted encoder.
This way is claimed to prevent from any data-leakage. However, this seems to often be omitted during cross-validation. Let's suppose I am performing cross validation on the aforementioned train set. If I encode train set and then perform cross-validation it doesn't really mimic the steps above. Shouldn't the encoding be performed "within" cross-validation then? For example, assuming that we perform 5-fold cross-validation, shouldn't we fit the encoder on 4 folds and transform on 5th fold in each cross-validation step? I believe it is what's usually done in target encoding, but not really with label or one-hot encoding.
My questions therefore are:
- Am I right about the necessity to fit the encoder on 4 folds and not on the 5th validation fold in each cross-validation step if we really want to prevent overfitting?
- If not, why is it really necessary to perform all 3 steps mentioned before while dealing with train and test (hold-out) set?