How to handle columns with categorical data and many unique values

Question

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution? Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

Answer 1

For categorical columns, you have two options :

1. Entity Embeddings
2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks

• On learning embeddings for categorical data using Keras

• Google Developers > Machine Learning > Embeddings: Categorical Input Data

• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner