I am working on a sequence prediction problem using Keras. My end-goal is to have my input be several features, one being a categorical variable, that I have used 1 hot encoding for, and another which is a numeric variable, for which I would like to predict the next time step(s) for BOTH variables (any other variables just as inputs).

My first attempt was to eliminate the numeric and just use the sequence of "words" (not really words, but essentially the same concept here) to learn model that could predict the rest of the sequence, given partially complete "sentences". I have built a working model that attempts this, with decent results.

Model thus far:

def LSTMModel(timestep_len,hidden_size,vocab_size):
    model = Sequential()
    model.add(Masking(mask_value=0.0, input_shape=(timestep_len,num_of_features)))
    model.add(LSTM(hidden_size, return_sequences=True))
    return model

input size to masking: (None, 27, 1)

output size from softmax: (None, 27, 6997)

I pad the sequences to all be of the same length, which is why there is a Masking layer.

I an stuck on how I can incorporate a numeric variable in with this. How do I need to adjust my architecture to accommodate for this? I realize this current one will not work for what I'm trying to do. This was just a simple first pass approach. Do I need a custom loss function for each variable and then add them together some how? My input is size (batch_size, max_seq_len, 1), where the 3rd dimension is a unique integer encoding for the "word". My input Y is of size (batch_size,max_seq_len,vocab_size) where vocab _size is the one-hot vector representation of the t+1 timestep. How do I extend this properly to add in another variable? How can I use a Softmax on part of the output of a dense layer? Is that even the path I should be thinking?

I am having not much luck finding good resources for this type of sequence prediction. Everything seems to be more simple and restricted.

Thanks for any help!


1 Answer 1


Basically every answer I was looking for was exampled and explained in this tutorial. Absolutely great resource for trying to understand how to model multi-output networks. This one goes through a lengthy walkthrough of a multi-output CNN architecture using Keras functional API. It only took me about three weeks to stumble upon, however.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.