Recurrent neural network multiple types of input Keras

Question

For a project I want to use recurrent neural networks, however my knowledge on this subject is still somewhat limited. I do have some experience with convolutional nets and traditional neural networks. I need to predict a probability distribution over one of the inputs of the next step. Most of my sequences are relatively short, with some big outliers in there. There is more than enough data so that should not be an issue.

I'm using Keras for this task, using either LSTM or GRU layers.

My input exists of sequences of categorical input $x_A$, $x_B$ and a few numerical values $x_1$ to $x_{11}$. I'm trrying to predict $x_A$ of the last step that is not trained on. For the high cardinality categorical features I'm using an embedding layer to map them to a dense space and then merge the three different types of input into one layer. My issue is merging them together before going to the softmax layer to do the one-hot encoding prediction of the last step. I cannot seem to get the inputs to match. I pad the shorter sequences to 10 steps plus the label and truncate the longer sequences cutting off the start. I've attempted both the Graph and the Sequential interface.

def build_graph_model(n_A, n_B, max_length, batch_size):
    g = Graph()
    g.add_input(name='A', input_shape=(max_length, ), dtype='int')
    g.add_node(Embedding(n_A, 32, input_length=max_length, mask_zero=True), name='embedding_A', input='A')
    g.add_input(name='B', input_shape=(max_length, ), dtype='int')
    g.add_node(Embedding(n_B, 32, input_length=max_length, mask_zero=True), name='embedding_B', input='B')
    g.add_input(name='rest', input_shape=(max_length, 11))
    g.add_node(Dense(30, activation='relu'), name='rest_dense', input='rest')
    g.add_node(LSTM(100), name='lstm', inputs=['embedding_A', 'embedding_B', 'rest_dense'])
    g.add_node(Dense(n_dest, activation='softmax'), name='softm', input='lstm')
    g.add_output(name='output', input='softm')
    g.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

This gives an assertion error for having the wrong input dimensions. If I remove the $x_1$ to $x_11$ thing completely it works. If I change the Dense layer into TimeDistributed(Dense) I still get the same error although I think that should work. Inserting the rest input directly into the merge doesn't work because the merge cannot be the first layer. I'm uncertain how to fix this issue.

EDIT: This guy asks the same question but much more concise (https://groups.google.com/forum/#!topic/keras-users/zCuS4xdv5QY)

Answer 1

Kera's functional API allows arbitrary models to be combined like you are describing.

One option is to define a model very similar to the documentation linked above. First, define a "main input" that is categorical, and then, define an "auxiliary input" that is numerical. Both of these sub-models can be merged into a single network with additional layers.