2
$\begingroup$

While doing target encoding it can happen that two categories have the same target mean.

This is bad because there will be no difference in the new feature in it and we will lose some information.

Also, this is potentially harmful to the model, choosing this split in the feature can produce some incongruences.

Is there any way to fix this problem?

Improve this question
$\endgroup$
3
  • $\begingroup$ This shouldn't be considered a problem. But if you do consider it a problem, you can put the categories themselves as a separate feature, probably after applying ordinal encoding to them. $\endgroup$
    – M. Saif
    Mar 8, 2021 at 12:34
  • 1
    $\begingroup$ The authors of the method argue that "An effective [encoding] approach is to group values that exhibit similar target statistics". They say their method makes "categorical values that exhibit similar statistics with respect to the target have a similar [numerical] representation". Thus,t if 2 categories return the same target estimates, it is a desirable effect. More details here: Micci-Barreca D. “A Preprocessing Scheme for High-Cardinality Categorical Attributes in Classification and Prediction Problems”. ACM SIGKDD Explorations Newsletter, 2001. dl.acm.org/citation.cfm?id=507538 $\endgroup$
    – Sole G
    Apr 25, 2022 at 20:27
  • $\begingroup$ Thanks, @SoleG Micci-Barreca paper is a classic in target encoding methods. Is expected that two categories with the same mean behave similarly, but this might not occur due to introducing a bias in the pre-processing. arxiv.org/abs/2201.11358 In this paper we present two types of bias that can be induced while encoding categorical attributes $\endgroup$
    – Carlos Mougan
    Apr 25, 2022 at 21:10

1 Answer 1

Reset to default
1
$\begingroup$

I assume tree-based models in this answer

Bayensian Mean Encoding can help

The main problem is that the model will be unable to split between the merged categories. So you implicitly accept that there is no interaction between the two merged categories and other variables. (succession of splits will be the same for cat1 and cat2)

If you want to enable the tree to split anywhere (because you suspect interactions), you have to create a space between the values of cat1 and cat2 for the tree to split. Bayesian Mean Encoding can help you. It takes the frequency of the categories into accounts in the calculation of the target means.

The formula is the following :

\begin{equation} \mu = \frac{n * \bar{x} + m*w}{n+m} \end{equation} where :

  • $\mu$ is the category mean
  • $n$ is the frequency of this category
  • $\bar{x}$ is the target mean for this category
  • $m$ is the weight you want to give to the overall mean
  • $w$ is the overall mean

nb: with small $m$ you will have a result very similar to simple target encoding while moving slightly the means (enough to enable the split). The main goal of Bayesian Mean is to limit overfitting, which is also a problem when doing target encoding.

There is no point of doing that

The idea behind target encoding is to make the assumption that the category has no interaction with other variables. So you convert the categorical space (high dimensional) into a simple continuous space, where only the value counts. If you think that the algorithm might confuse two categories and that this is bad, it means target encoding is not suited. If different categories should follow a different decision path, then it means there are interactions between the categorical variable and the others.

If you want every category to have different decisional paths why don't you keep categories? Because implicitly, the tree will compute mean of the splits (explained in this video: https://www.youtube.com/watch?v=g9c66TUylZ4)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Even thought this answer minimizes the chances of this happening. It does not resolve the main problem. What happens if two classes (even after adding weight, frequency to it) get the same value. $\endgroup$
    – Carlos Mougan
    Apr 26, 2020 at 7:38
  • 1
    $\begingroup$ This is true. I can imagine creating a custom "OrdinalEncoder" that assigns labels with respect to the value of the group target. It would have the same effect as perfecting my Bayesian Encoder to distinguish each category. However, I edited my answer to explain why it does not make sense to do this. $\endgroup$
    – Rusoiba
    Apr 26, 2020 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.