You can create an embedding (dense vector) space for your categorical variables. Many of you are familiar with word2vec and fastext, which embed words in a meaningful dense vector space. Same idea here-- your categorical variables will map to a vector with some meaning.
From the Guo/Berkhahn paper:
Entity embedding not only reduces memory usage and speeds up neural
networks compared with one-hot encoding, but more importantly by
mapping similar values close to each other in the embedding space it
reveals the intrinsic properties of the categorical variables. We
applied it successfully in a recent Kaggle competition and were able
to reach the third position with relative simple features.
The authors found that representing categorical variables this way improved the effectiveness of all machine learning algorithms tested, including random forest.
The best example might be Pinterest's application of the technique to group related Pins:
The folks at fastai have implemented categorical embeddings and created a very nice blog post with companion demo notebook.
Additional Details and Explanation
A neural net is used to create the embeddings i.e. assign a vector to each categorical value. Once you have the vectors, you may use them in any model which accepts numerical values. Each component of vector becomes an input variable. For example, if you used 3-D vectors to embed your categorical list of colors, you might get something like: red=(0, 1.5, -2.3), blue=(1, 1, 0) etc. You would use three input variables in your random forest corresponding to the three components. For red things, c1=0, c2=1.5, and c3=-2.3. For blue things, c1=1, c2=1, and c3=0.
You don't actually need to use a neural network to create embeddings (although I don't recommend shying away from the technique). You're free to create your own embeddings by hand or other means, when possible. Some examples:
- Map colors to RGB vectors.
- Map locations to lat/long vectors.
- In a U.S. political model, map cities to some vector components representing left/right alignment, tax burden, etc.