3
$\begingroup$

Recently I've been thinking about the proper use of encoding within cross-validation scheme. The customarily advised way of encoding features is:

  1. Split the data into train and test (hold-out) set
  2. Fit the encoder (either LabelEncoder or OneHotEncoder) on the train set
  3. Transform both the train and test set using fitted encoder.

This way is claimed to prevent from any data-leakage. However, this seems to often be omitted during cross-validation. Let's suppose I am performing cross validation on the aforementioned train set. If I encode train set and then perform cross-validation it doesn't really mimic the steps above. Shouldn't the encoding be performed "within" cross-validation then? For example, assuming that we perform 5-fold cross-validation, shouldn't we fit the encoder on 4 folds and transform on 5th fold in each cross-validation step? I believe it is what's usually done in target encoding, but not really with label or one-hot encoding.

My questions therefore are:

  1. Am I right about the necessity to fit the encoder on 4 folds and not on the 5th validation fold in each cross-validation step if we really want to prevent overfitting?
  2. If not, why is it really necessary to perform all 3 steps mentioned before while dealing with train and test (hold-out) set?
$\endgroup$

1 Answer 1

1
$\begingroup$

You're right, the encoding step itself can be a source of data leakage and normally it should be done inside the CV loop using only the current training set, as you describe.

The reason is indeed the one you mention in the comment: if there is a class label or a feature category which doesn't appear by chance in a particular training set during CV, the model is not supposed to know that this class/category even exists.

In general I would think that this issue can only decrease the performance on the test set, so it's probably not as serious as other kinds of data leakage. Still, it's definitely a cleaner experimental design to encode using only the training set.

A closely related issue in NLP is when the system is not designed to deal with out-of-vocabulary (OOV) words: if all the words in both the training and test set are encoded (same mistake), then it wrongly looks as if any text can be fully encoded, potentially causing to bad surprises later.

That being said, it's usually a good idea to discard rare features or label values, and if this is done then the result should be the same using either the proper method or the sloppy one.

$\endgroup$
3
  • 1
    $\begingroup$ Could you elaborate on how exactly it decreases the performance on the test set? I mean, from my understanding, when all the feature's values are 0 in the train set this column is effectively ignored by decision-tree-based models (no splits are being made on this particular feature as you can't really split one number). On test set, value other than zero appears, but the model isn't instructed to split on that feature, so it ignores it again. The situation might be different with linear/logistic regression models where column of 0 is collinear with any other feature. Am I missing something? $\endgroup$
    – jakes
    Aug 18, 2020 at 5:37
  • 1
    $\begingroup$ @jakes that would depend on the learning algorithm indeed. I imagine that with something like linear regression the feature would be assigned a coefficient like any other feature, so a different feature value in the test set could mess up the prediction. Also I'm not sure how to consider the more serious case of a label missing in the training set from the point of view of evaluation, but it can't be good. I think in most cases performance would be the same, I can imagine that it could decrease in some cases, but I cannot imagine any case where it would increase -> this is the most ... $\endgroup$
    – Erwan
    Aug 18, 2020 at 9:44
  • 1
    $\begingroup$ ... serious problem usually caused by data linkage, overestimating performance. $\endgroup$
    – Erwan
    Aug 18, 2020 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.