Target encoding with cross validation

Question

I am trying to understand this way of target (mean/impact/likelihood) encoding using (two-level) cross validation.

It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;

Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.

So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.

It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)

As far as I understand, the motivation of this approach is that: target encoding requires the knowledge of output, which is not available on the test set. So if we use the means obtained from the whole train set and apply on test set, that may cause overfitting. So instead, we will use other values derived from its subset.

I found some discussions from this post but have trouble of understanding the following points:

1) It seems to me that the his second-level CV is nothing but taking the average of the whole #2-#20 fold. So basically, this is just one-level cross validation, where instead of using the mean of #1 fold, we use that of #2-#20 fold as the mean value for #1 fold. Am I missing something here?

2) Once we obtain the means of all 20 folds, what will we do next? If we average, this is again nothing but taking average of all train set.

Answer 1

Your question 1:

You are close but not exactly.

Each time you take the mean, you include only the samples with the given value, say, A, of the categorical feature.

You partition #2-#20 folds into 10 equal folds; however, once you consider only the samples with the category value A, their numbers may be different in these 10 folds.

Say, these numbers are n1,...,n10.

Then the mean of all but the first fold assigns the weight of 0 to the samples in the first fold and the weight of 1/(n2+...+n10) to each sample in all other folds.

The mean of all but the second fold assigns the weight of 0 to the samples in the second fold and the weight of 1/(n1+n3+n4+...+n10) to each sample in all other folds, etc.

Taking the average of these means yields the weighted average of the target with the weights of the samples being the averages of the above weights.

These weights will be slightly different for samples in different folds.

Thus, we get the weighted average of #2-#20 folds but with slightly different weights obtained by this random procedure.

You probably want to understand what is the point of this: how is it better than just taking the plain average over folds #2-#20?

This process adds randomality to the values of the encoding. (The randomality can be measured, e.g., by the standard deviation of the encoding values for the category value A.)

This randomality of the encoding values may prevent the algorithm from learning the relationship between the encoding and the target in case this relationship holds only in the training set, i.e., reduce overfitting.

This randomality depends on two parameters:

1. The width of the distribution of the target value for the samples having the category value A. (The width may be measured, e.g., by the standard deviation.) Wider distribution in these target values results in greater randomality (wider distribution) of the encoding values.

2. The number of samples having the category value A: more samples result in smaller randomality of the encoding values.

One can compare this way with other ways of increasing the randomality of the encoding values. The simplest way is to do single cross-validation but with less than 20 folds. This will increase the randomality of the encoding values but it will mainly depend on the width of the distribution of the target values and less on the the number of samples in the category A. This could be the reason for this double cross-validation.

Your question 2:

As you said, the point is to use for each sample the average of the target values that does not include this sample.

For a sample in the fold #1 with the category value A, the value of the mean target encoding is the (weighted) mean of the target values over all samples in the folds #2-#20 with the category value A (calculated as above).

This means that the values of the encoding of A will be different in different folds.

The post you quoted, does not mention the test set. Other posts suggest that on the test set you take the average of the target values of all samples that have category A in the entire train set.

Answer 2

@Baruch Youssin I am not sure I understand why the variability introduced by the k-fold target mean encoding calculation decreases overfitting.

I understand that by applying k-fold target mean encoding instead of mean encoding you avoid data leakage (i.e. using information about a specific row of Y to calculate a feature X).

But then assuming no data leakage and since in your test set you will use a single value to represent a category (the mean of your k target mean encodings calculated in the k-folds of the training set) in the feature X, then I understand that what you would only care about would be that the relationship you have fit with your model between Y and feature X at the value of X used to represent the category in the test set, remains the same within both your training set and your test set. Not sure if I am missing something else here.