LSTM not learning with extra nontemporal data added after LSTM layer - Keras

Question

I have three different inputs I would like to send into my LSTM - the sequence of "words", extra temporal information, as well as extra nontemporal information.

Following Adam Sypniewski excellent answer to this question (Adding Features To Time Series Model LSTM) I came with this architecture:

word_input = Input(shape=(sequence_length,))
temporal_metadata = Input(shape=(sequence_length, 1))
nontemporal_metadata = Input(shape=(nontemporal_num_features,))

embedding = Embedding(input_dim=vocab_size, output_dim=100)(word_input)
temporal_info = concatenate(inputs=[embedding, temporal_metadata])
rnn = LSTM(50, dropout = .5, recurrent_dropout=.5)(temporal_info)
all_info = concatenate(inputs=[rnn, nontemporal_metadata])
dense_1 = Dense(10, activation='relu')(all_info)
dropout_1 = Dropout(.5)(dense_1)
dense_2 = Dense(1, activation='sigmoid')(dropout_1)

model = Model([word_input, temporal_metadata, nontemporal_metadata], dense_2)
model.compile(loss='binary_crossentropy', optimizer='adam'
model.fit(x=[my_words, my_temporal, my_nontemporal], y = my_y, 
     epochs = 30, 
     validation =([my_words_test, my_temporal_test, my_nontemporal_test], y_test))

This does not learn, and gets an AUC of 0.5

However if I don't add the nontemporal data and change the above code to

word_input = Input(shape=(sequence_length,))
temporal_metadata = Input(shape=(sequence_length, 1))

embedding = Embedding(input_dim=vocab_size, output_dim=100)(word_input)
temporal_info = concatenate(inputs=[embedding, temporal_metadata])
rnn = LSTM(50, dropout = .5, recurrent_dropout=.5)(temporal_info)
dense_1 = Dense(10, activation='relu')(rnn)
dropout_1 = Dropout(.5)(dense_1)
dense_2 = Dense(1, activation='sigmoid')(dropout_1)

model = Model([word_input, temporal_metadata], dense_2)
model.compile(loss='binary_crossentropy', optimizer='adam'
model.fit(x=[my_words, my_temporal], y = my_y, 
     epochs = 30, 
     validation =([my_words_test, my_temporal_test], y_test))

It works great. Can anyone please help me find what is going wrong in the first code sample that would make the model unable to learn. Thanks!

Answer 1

if anyone stumbles upon this - it seemed like my dimension of nontemporal data was too high and sparse. After filtering only for the most common/important variables in the nontemporal data, the model learned.