Applying mean encoding before or after splitting into train and test set

Question

I have a dataset of 50000 observations with columns of high cardinality. The best way to encode them is with mean encoding, then to use regularization. I will use CV rather than smoothing. But when I see people use it, they use it on train and test set.

Should I first split my dataset into train and test set and then encode or can I encode directly from the beginning on my full dataset?

If I should split the data into train and test set first, can someone tell me why?

Answer 1

The purpose of having a test set or a validation set is to be able to check the performance of your model on data it has not seen before. If you perform feature engineering with the test data present you will get a data leakage. That happens when give your model information about your test data during training.

It is especially bad when doing target encoding with the label mean since it will give your model information about the distribution of the labels in the test set. The effect is that you will get an overly optimistic test score that will not reflect the performance on new truly unseen data.

Answer 2

In addition to Simon's answer: You need to do mean encoding after splitting the data. Note that you should not mean encode them separately. You need to get the means from the training set and map it to test set. This would not totally remove data leakage though.

Simple mean encoding:

means = X_tr.groupby(col).target.mean()
train_new[col+'_mean_target'] = train_new[col].map(means)
val_new[col+ '_mean_target'] = val_new[col].map(means)

Sometimes you will end up with missing values in the test set because of a mismatch between train and test groups. In this case, you might need to fill missing values with the mean of the means. Something like this should work:

train_new[col+ '_mean_target'].fillna(means.mean(), inplace = True)