I am working on a sequence prediction problem using Keras. My end-goal is to have my input be several features, one being a categorical variable, that I have used 1 hot encoding for, and another which is a numeric variable, for which I would like to predict the next time step(s) for BOTH variables (any other variables just as inputs).
My first attempt was to eliminate the numeric and just use the sequence of "words" (not really words, but essentially the same concept here) to learn model that could predict the rest of the sequence, given partially complete "sentences". I have built a working model that attempts this, with decent results.
Model thus far:
def LSTMModel(timestep_len,hidden_size,vocab_size):
model = Sequential()
model.add(Masking(mask_value=0.0, input_shape=(timestep_len,num_of_features)))
model.add(LSTM(hidden_size, return_sequences=True))
model.add(TimeDistributed(Dense(vocab_size)))
model.add(Activation('softmax'))
return model
input size to masking: (None, 27, 1)
output size from softmax: (None, 27, 6997)
I pad the sequences to all be of the same length, which is why there is a Masking layer.
I an stuck on how I can incorporate a numeric variable in with this. How do I need to adjust my architecture to accommodate for this? I realize this current one will not work for what I'm trying to do. This was just a simple first pass approach. Do I need a custom loss function for each variable and then add them together some how? My input is size (batch_size, max_seq_len, 1), where the 3rd dimension is a unique integer encoding for the "word". My input Y is of size (batch_size,max_seq_len,vocab_size) where vocab _size is the one-hot vector representation of the t+1 timestep. How do I extend this properly to add in another variable? How can I use a Softmax on part of the output of a dense layer? Is that even the path I should be thinking?
I am having not much luck finding good resources for this type of sequence prediction. Everything seems to be more simple and restricted.
Thanks for any help!
