Hi everybody and thanks in advance for those who will help me for this problem. I have multiple data regarding patients involved in a clinical trial and my goal is to predict their death/non death. These data are composed by two datasets , the first is static (age, gender ecc) and the second is dynamic (analysis results collected during multiple sessions like %hematocrit, %sodium, blood pressure and so on). I know that if I had only static data I would have used any ML classification algorithm and of course in case of only dynamic data the "right" way was to build a LSTM but with both of them I'm a bit confused. Which kind of model can I build in this case? I was thinking at two models where the output of the first one (static) becomes the input of the second one (dynamic). Is it a good idea? Thank you.


1 Answer 1


What you are referring to is called a multi-input model and can be esaily built in most deel learning frameworks. The idea is to have both types of data as separate inputs, then use specific layers depending their types (recurrent layers to sequence data, CNN to images, and so on...) to later on concatenate them together.

If you can use Keras, there is the functional Api which is specialy well suited for the task at hand. An example of code (based the example given in the documentation) for your problem could be:

from keras.layers import Input, Embedding, LSTM, Dense, merge
from keras.models import Model

# headline input: meant to receive sequences of 100 integers, between 1 and 10000.
# note that we can name any layer by passing it a "name" argument.
main_input = Input(shape=(100,), dtype='int32', name='main_input')

# this embedding layer will encode the input sequence
# into a sequence of dense 512-dimensional vectors.
x = Embedding(output_dim=512, input_dim=10000, input_length=100)(main_input)

# a LSTM will transform the vector sequence into a single vector,
# containing information about the entire sequence
lstm_out = LSTM(32)(x)

#At this point, we feed into the model our auxiliary input data by concatenating it with the LSTM output:

auxiliary_input = Input(shape=(5,), name='aux_input')
x = merge([lstm_out, auxiliary_input], mode='concat')

# we stack a deep fully-connected network on top
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)

# and finally we add the main logistic regression layer
main_output = Dense(1, activation='sigmoid', name='main_output')(x)

#This defines a model with two inputss:
model = Model(input=[main_input, auxiliary_input], output=main_output)

#Then compite and train
model.compile(optimizer='rmsprop', loss='binary_crossentropy')
model.fit([headline_data, additional_data], labels,
          nb_epoch=50, batch_size=32)

In your case, the dynamic data would be the headline_input and your static data the auxiliary_input. The model will take both, apply the recurrent layer to the former and concatenate them to pass the union through the dense layers.

Of course, many of these parameters will depend on your data, but at least this example will give you an idea on how to build such model.

There is also this interesting project conditionall RNN which is also meant for this purposes. Worth taking a look.

  • $\begingroup$ Thank you very much for the answer, I'll giver a try. In my case shouldn't be the headline_input the static data and auxiliary_input the dynamic data? That's because the Embedding is mostly used with text and not with time series, isn't it? $\endgroup$
    – Rick0
    Commented Jun 2, 2020 at 11:03
  • 1
    $\begingroup$ Although there is no clear disadvantage in using them, yes I guess you could just remove the embedding layer. BUt still, make sure your sequence data is passed to the LSTM, therefore in the example it should correspond to the main_input but without the embedding layer → main_input = Input(shape=(100,), dtype='int32', name='main_input') and then lstm_out = LSTM(32)(main_input). aux_input goes straight to the dense layers, so no dynamic data there ;) $\endgroup$
    – TitoOrt
    Commented Jun 2, 2020 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.