1
$\begingroup$

I have three different inputs I would like to send into my LSTM - the sequence of "words", extra temporal information, as well as extra nontemporal information.

Following Adam Sypniewski excellent answer to this question (Adding Features To Time Series Model LSTM) I came with this architecture:

word_input = Input(shape=(sequence_length,))
temporal_metadata = Input(shape=(sequence_length, 1))
nontemporal_metadata = Input(shape=(nontemporal_num_features,))

embedding = Embedding(input_dim=vocab_size, output_dim=100)(word_input)
temporal_info = concatenate(inputs=[embedding, temporal_metadata])
rnn = LSTM(50, dropout = .5, recurrent_dropout=.5)(temporal_info)
all_info = concatenate(inputs=[rnn, nontemporal_metadata])
dense_1 = Dense(10, activation='relu')(all_info)
dropout_1 = Dropout(.5)(dense_1)
dense_2 = Dense(1, activation='sigmoid')(dropout_1)

model = Model([word_input, temporal_metadata, nontemporal_metadata], dense_2)
model.compile(loss='binary_crossentropy', optimizer='adam'
model.fit(x=[my_words, my_temporal, my_nontemporal], y = my_y, 
     epochs = 30, 
     validation =([my_words_test, my_temporal_test, my_nontemporal_test], y_test))

This does not learn, and gets an AUC of 0.5

However if I don't add the nontemporal data and change the above code to

word_input = Input(shape=(sequence_length,))
temporal_metadata = Input(shape=(sequence_length, 1))

embedding = Embedding(input_dim=vocab_size, output_dim=100)(word_input)
temporal_info = concatenate(inputs=[embedding, temporal_metadata])
rnn = LSTM(50, dropout = .5, recurrent_dropout=.5)(temporal_info)
dense_1 = Dense(10, activation='relu')(rnn)
dropout_1 = Dropout(.5)(dense_1)
dense_2 = Dense(1, activation='sigmoid')(dropout_1)

model = Model([word_input, temporal_metadata], dense_2)
model.compile(loss='binary_crossentropy', optimizer='adam'
model.fit(x=[my_words, my_temporal], y = my_y, 
     epochs = 30, 
     validation =([my_words_test, my_temporal_test], y_test))

It works great. Can anyone please help me find what is going wrong in the first code sample that would make the model unable to learn. Thanks!

$\endgroup$
1
$\begingroup$

if anyone stumbles upon this - it seemed like my dimension of nontemporal data was too high and sparse. After filtering only for the most common/important variables in the nontemporal data, the model learned.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.