I'm currently working on a project where I'm using an LSTM to learn and predict sequences of categorical data.
My dataset consists of variable-length sequences of items $s_i = [x_{i_0}, x_{i_1}, ..., x_{i_{-1}}]$. Each item $x$ belongs to one of 190,000 categories. Therefore, my initial approach has been to encode each $x$ with a one-hot vector representing its category.
However, using one-hot encoding makes the computational cost very high, and I want to find a more compact way of representing the categories before training the model.
In information theory, the base-2 logarithm of a number indicates how many bits are necessary to represent a number. In this case, log2(190000) =~ 18, which means that 18 bits are needed to represent each category.
So, my question is, what are good approaches to represent the categories using lower-dimensional vectors?
For example, I could use 18-dimensional binary vectors, where the index of each category is encoded in binary format. If I understand correctly, this approach would introduce a false additive relationship, but it might be worth experimenting with.
Alternatively, I am considering passing the one-hot vectors through a linear layer with 190k inputs and 18 outputs. The idea here is that the linear layer does not produce zero outputs but small floats. However, I don't know how well this approach can encode the categorical information.
Do these approaches make sense? Alternatively, what might be better ways to achieve what I'm looking for?
I appreciate any advice or thoughts on this approach. Thanks!