Handle multiple categorial features in character level RNN

Question

I am working on a fantasy name generator and I have 2 auxiliary categorical features (gender and race). I initially tried concatenating their one hot tensors directly into the input tensor (I think it's the most popular approach), but the model failed to differentiate between continues and categorical features (ignores the categorical features).

I read several similar questions and the closest ones I found were this and this, which suggested first combining these features via a Dense layer(or multiple layers) and then concatenate Dense layer's output with input tensor (of names). One alternative approach I have found is using thi Are there any other alternate approaches ? also one question I have about this approach is that here too, ultimately the auxiliary features are concatenated with input tensor. Why should the model not ignore features with this approach ?

Answer 1

The usual approach to encode discrete stuff in NLP is to use an embedding layer, which saves you a huge matrix multiplication if you were using one-hot input representations.

Once you have your discrete feature encoded with an embedding layer, you have two options to merge it with the rest of the data: concatenate it or add it.

If you concatenate, you have to assign dimensionalities to each part a priori and the more space you allocate for a feature, the less space there is for the rest. This approach was used for "factored neural machine translation" (see the original scientific article) to encode morphological features into the associated words (e.g. verbal tense for verbs, number and gender for nouns, etc).

On the other hand, if you add your embedded feature to the rest of the data, it means that the embedding vector for the feature is of the same dimensionality as the rest of the data. This approach is used in transformers to add positional encodings to token encodings:

I suggest you go with this approach. In my experience gives better results in general.

Of course, either way the features can be ignored during training if they are not useful to improve the loss.