While doing target encoding it can happen that two categories have the same target mean.

This is bad because there will be no difference in the new feature in it and we will lose some information.

Also, this is potentially harmful to the model, choosing this split in the feature can produce some incongruences.

Is there any way to fix this problem?

  • $\begingroup$ This shouldn't be considered a problem. But if you do consider it a problem, you can put the categories themselves as a separate feature, probably after applying ordinal encoding to them. $\endgroup$ – M. Saif Mar 8 at 12:34

I assume tree-based models in this answer

Bayensian Mean Encoding can help

The main problem is that the model will be unable to split between the merged categories. So you implicitly accept that there is no interaction between the two merged categories and other variables. (succession of splits will be the same for cat1 and cat2)

If you want to enable the tree to split anywhere (because you suspect interactions), you have to create a space between the values of cat1 and cat2 for the tree to split. Bayesian Mean Encoding can help you. It takes the frequency of the categories into accounts in the calculation of the target means.

The formula is the following :

\begin{equation} \mu = \frac{n * \bar{x} + m*w}{n+m} \end{equation} where :

  • $\mu$ is the category mean
  • $n$ is the frequency of this category
  • $\bar{x}$ is the target mean for this category
  • $m$ is the weight you want to give to the overall mean
  • $w$ is the overall mean

nb: with small $m$ you will have a result very similar to simple target encoding while moving slightly the means (enough to enable the split). The main goal of Bayesian Mean is to limit overfitting, which is also a problem when doing target encoding.

There is no point of doing that

The idea behind target encoding is to make the assumption that the category has no interaction with other variables. So you convert the categorical space (high dimensional) into a simple continuous space, where only the value counts. If you think that the algorithm might confuse two categories and that this is bad, it means target encoding is not suited. If different categories should follow a different decision path, then it means there are interactions between the categorical variable and the others.

If you want every category to have different decisional paths why don't you keep categories? Because implicitly, the tree will compute mean of the splits (explained in this video: https://www.youtube.com/watch?v=g9c66TUylZ4)

  • $\begingroup$ Even thought this answer minimizes the chances of this happening. It does not resolve the main problem. What happens if two classes (even after adding weight, frequency to it) get the same value. $\endgroup$ – Carlos Mougan Apr 26 '20 at 7:38
  • 1
    $\begingroup$ This is true. I can imagine creating a custom "OrdinalEncoder" that assigns labels with respect to the value of the group target. It would have the same effect as perfecting my Bayesian Encoder to distinguish each category. However, I edited my answer to explain why it does not make sense to do this. $\endgroup$ – Rusoiba Apr 26 '20 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.