3
$\begingroup$

I have a dataset of 50000 observations with columns of high cardinality. The best way to encode them is with mean encoding, then to use regularization. I will use CV rather than smoothing. But when I see people use it, they use it on train and test set.

Should I first split my dataset into train and test set and then encode or can I encode directly from the beginning on my full dataset?

If I should split the data into train and test set first, can someone tell me why?

$\endgroup$
3
$\begingroup$

The purpose of having a test set or a validation set is to be able to check the performance of your model on data it has not seen before. If you perform feature engineering with the test data present you will get a data leakage. That happens when give your model information about your test data during training.

It is especially bad when doing target encoding with the label mean since it will give your model information about the distribution of the labels in the test set. The effect is that you will get an overly optimistic test score that will not reflect the performance on new truly unseen data.

| improve this answer | |
$\endgroup$
  • $\begingroup$ alright , i get it , so splitting then mean encoding , thank you $\endgroup$ – Dimi May 19 '19 at 21:06
1
$\begingroup$

In addition to Simon's answer: You need to do mean encoding after splitting the data. Note that you should not mean encode them separately. You need to get the means from the training set and map it to test set. This would not totally remove data leakage though.

Simple mean encoding:

means = X_tr.groupby(col).target.mean()
train_new[col+'_mean_target'] = train_new[col].map(means)
val_new[col+ '_mean_target'] = val_new[col].map(means)

Sometimes you will end up with missing values in the test set because of a mismatch between train and test groups. In this case, you might need to fill missing values with the mean of the means. Something like this should work:

train_new[col+ '_mean_target'].fillna(means.mean(), inplace = True)
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you a lot for the additional insight. $\endgroup$ – Dimi May 24 '19 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.