How to add multiple embeddings (layers) to LSTM layer

Question

The similar question was asked before here https://stackoverflow.com/questions/52627739/how-to-merge-numerical-and-embedding-sequential-models-to-treat-categories-in-rn/52629902#comment136040845_52629902, but I didn't understand a clear answer. I will use the author’s @GRS materials to explain.

I'm trying to solve the problem of classifying sequences of categorical features. For this task, the vector of the last hidden state from the LSTM layer is very often used. For each categorical variable of the sequence I define learnable layer nn.Embedding.

The question is which approach is better and more correct: create a separate lstm layer for each feature - categorical embedding layer (as in the picture), or combine all embeddings and transfer them (like one tensor) into a single lstm layer? The first approach I never see, but I don't understand, why it is not popular.

Thank you very much in advance

Answer 1

Let's consider the main approaches:

1. Embed each categorical feature, then LSTM on top. I think you want to opt for individual recurrent layers when the features are sort of mutually exclusive, i.e. the information contained in one feature is not beneficial for the others, and vice-versa. Also this method can be computationally slower since you have multiple recurrent layers, and maybe also more difficult to train (again, due to multiple BPTT.) The final step is then to aggregate (merge) all the features learned by the LSTMs for the last layer.
2. Merge the various embeddings, then recurrence. Otherwise you can think of combining the features first with a dense or even linear layer. In this way you can also reduce the dimensionality of the resulting embedding, which can be beneficial also to remove some redundant information. There are multiple ways to aggregate, like addition and multiplication (assuming same dimensionality of embeddings) or the usual concatenation, which is usually followed by a dense layer. Once the embeddings have been merged, you apply the LSTM on top followed by the final layer. Here you have a single recurrence that takes in a combination of all the embedded features, learning one single joint feature space.

Indeed, to find the best solution for your problem you should compare both approaches.

Answer 2

My opinionated answer is I don't see much point in having multiple LSTMs.

I would map all categorical features to an embedding. For the numerical features, I would use a MLP to map them to a vector of the same size as your embeddings. Then cat all those together into a sequence of embeddings and send that to a single LSTM layer.