How to handle a large number of categories in one column effectively in machine learning?

Question

In my dataset one column has a large no of categories. For model training or building we need to convert these categories to numeric. There are several methods we can use to convert labels to numeric, one-hot encoding and label encoders, but here my problem is that the categories are not ordinal (no need to preserve order). We can use one-hot encoding, but it creates a large number of dimensions. In this case, which method do I need to choose for converting the categories to numeric effectively? It is not supposed to affect the dimensions of the model.

Please advise on this case. Thanks in advance.

Answer 1

Target encoding calculated using an appropriate cross-validation strategy can also be powerful for high-cardinality categorical features.

In some instances, frequency encoding can also be useful.

Answer 2

You could use Binary or BaseN encoding because they are commonly used with high dimensional nominal categories. Because binary or baseN encoding encodes the categories into ordinal numbers and then into binary or base of N respectively in an effective way. They are so efficient in terms of complexity and the dimensionality. I recommend you to read this for more information here. Moreover, you can find here a different number of encodings with the comparison between them here

Note: BaseN is a general encoding and other encodings are a special case of BaseN. For instance, if N=1, this is one-hot encoding. If N=2, this is a binary encoding, and so on.

Answer 3

The standard method to encode a categorical variable is one hot encoding. Replacing categories with numbers (ordinal encoding) would certainly introduce errors in the model because it would rely on numerical comparisons which are meaningless with categorical values.

The high number of dimensions can be a problem if the number of instances is too low and/or the variable has too many rare values. The risk is overfitting: the model would rely on values which happen by chance because it doesn't have a large enough representative sample. In general the solution is to simplify the data: replace rare values (those which occur less than $n$ times in the training set) with a special value other for example. Note that you can adjust the number of dimensions by varying the threshold $n$. It's very likely that there are many rare values and a smaller number of frequent values, therefore this method reduces the number of dimensions quickly. Note that the threshold $n$ can be determined by parameter tuning, but in this case you need a separate validation set (different from the final test set).

Note that you must always define the encoding using only the training data, then apply the predefined encoding on the test set. If the test set contains a value which doesn't exist in the training set, it should similarly be replaced with the special value other.

---

[edit] Note that all of the above is very generic advice about the potential problems and possible solutions for this case. As usual, it depends a lot on the specifics of the task and data. The only way to find the optimal method is to experiment (Thanks to Sammy for reminding me to mention this).