1
$\begingroup$

From Kaggle's intermediate machine learning tutorial, it was stated that

for each column, we randomly assign each unique value to a different integer. This is a common approach that is simpler than providing custom labels; however, we can expect an additional boost in performance if we provide better-informed labels for all ordinal variables.

Here's what I understood:

If I had a column named place with the unique values being [first, second, third], then I would get better performance by encoding those as [1,2,3] compared to [2,1,3]. Is my understanding correct? If so, how does this lead to better performance? Since the integers are just used as a numeric placeholder for the unique values, does the ordering even matter as long as those integers can uniquely identify each value?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If I had a column named place with the unique values being [first, second, third], then I would get better performance by encoding those as [1,2,3] compared to [2,1,3]. Is my understanding correct?

Yes, your understanding is correct.

If so, how does this lead to better performance? Since the integers are just used as a numeric placeholder for the unique values, does the ordering even matter as long as those integers can uniquely identify each value?

There is a confusion between:

  • the actual meaning for a human of these integers in this particular case -> in this sense yes, they are just used as a numeric placeholder
  • how these values are interpreted by a ML algorithm, e.g. classifier -> the ordering does matter for the algorithm, it considers these values as numerical and treat them like any other feature.

The second point implies that the algorithm may use a condition v>=2, this could cause overfitting if the ordering doesn't make any sense.

For the record I don't understand the logic of this part of the tutorial: to me an ordinal variable has by definition an order which is assumed to be known, it doesn't make sense to use OrdinalEncoder on the wrong order. It looks like they are talking about using OrdinalEncoder on some categorical variable (sometimes done to avoid high number of features with one-hot encoding) or some variable of unknown type, maybe?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! For your last point, you are right. At one point, the tutorial mentioned that one-hot does not work well when the categorical variable takes on a large number of values. They discarded all columns with >=10 unique categorical values, then compared the performance between ordinal encoding and one-hot encoding. One-hot performed better in this case. $\endgroup$
    – Terrarium
    Commented Nov 8, 2022 at 16:00
  • $\begingroup$ @Terrarium ok thanks, it's similar to what I suspected. This is a matter of opinion actually, but personally I prefer to encode categorical variables as OHE even with large number (and 10 is not large imho). But it's true that if the number of values is really large it can cause problems (curse of dimensionality). In such a case I would rather try to remove the least frequent values first (replace with a default value), this is often enough. $\endgroup$
    – Erwan
    Commented Nov 8, 2022 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.