I am working with an LSTM nn built with Keras. I have the need to pass in a history of events as a single factor and I'm thinking of doing it with as a list. But I've never passed in a list in this manner - is that something that is possible with Keras? Or do I need to go through some sort of categorization process for this?

  • $\begingroup$ What do you mean by 'a single factor'? Would it be okay if you could pass in a matrix row by row? $\endgroup$ – Elias Strehle Mar 6 '18 at 13:11
  • $\begingroup$ @EliasStrehle I can shape the data in any form really. I can turn it into a matrix but the question still remains if an LSTM can accept a matrix as one of many inputs $\endgroup$ – I_Play_With_Data Mar 6 '18 at 13:34
  • $\begingroup$ @UnknownCoder You and others answering your question are focusing too much on the data structure aspect of a Keras model. What you need to ask yourself is, what does it mean to feed a history of events into a LSTM model, and does it make sense to do so. And does it improve model fit and generalization with this type of predictors. $\endgroup$ – horaceT Mar 13 '18 at 4:25
  • $\begingroup$ @horaceT I'm clear on the factors I need. Some of us are doing this on a professional basis and can't always reveal what we're working on. So, the question is intentionally obfuscated and the need for the history and other independent variables being part of the model should be taken as a given, not a point of discussion. $\endgroup$ – I_Play_With_Data Mar 14 '18 at 21:46

Your input matrix should have dimensions (# of instances, length of history of events, 1). For example, I have text data that is sequential and I will cut it up into sequences which are sequentially dependent. Each row in the matrix is the sequence, the history of events. My label for each of these is the next sequential letter, or event in your case.

The input $X$ has the following shape


(163717, 100, 1)

The LSTM model in Keras.

# define the LSTM model
model = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam') 
model.fit(X, y, epochs=20, batch_size=128, callbacks=callbacks_list)

$y$ is my targets. Can be any kind of classification.

If you have 1000 instances of these sequences of events which have 50 historical values then your matrix should have shape (1000, 50, 1)

If you want to add structured data you can always do this after the LSTM layers into the densely connected layer. You can have two models whos outputs are concatenated and then added together. This is a lot easier to do with the functional API of Keras as it is much more flexible and allows you to interact with the tensors more directly. For example

from keras.models import Sequential, Model
from keras.layers import Concatenate, Dense, LSTM, Input, concatenate
from keras.optimizers import Adagrad

seq_input = Input(shape=(X.shape[1], X.shape[2]))
seq_model = LSTM(256)(seq_input)
seq_model = Dropout(0.2)(seq_model)
seq_model = Dense(100, activation = 'softmax')(seq_model)

metadata_input = Input(shape=(8, ))
metamodel = Dense(10, )(metadata_input)

merged = concatenate([seq_model, metamodel])
merged = Dense(64, activation = 'softmax')(merged)
merged = Dense(y.shape[1], activation = 'softmax')(merged)

model = Model(inputs=[seq_input, metadata_input], outputs=merged)
model.compile(loss='categorical_crossentropy', optimizer='adam')

This example is as above. However, I add a metadata input of size 8 to the model after the LSTM layers. This will contribute in the output as it will embed the information from those features in the last few layers of the model.

  • $\begingroup$ I understand what you're saying and I appreciate the input, I have done NLP like this in the past too. However, the question is how do you make this just one of many other factors that you pass in? Coming back to your example, let's say you want to use that text, but also have other, structured factors, like ints & categories, etc. $\endgroup$ – I_Play_With_Data Mar 6 '18 at 15:55
  • $\begingroup$ I appended an answer to that more specific question. Sorry I misunderstood at first, $\endgroup$ – JahKnows Mar 6 '18 at 17:38

If I understand you correctly, you are wondering whether it is possible to have inputs where one element is a list or a matrix. It is not.

First, this would require a complete rewrite of all activation and decision functions in Keras. Take a linear activation for instance. Keras expects that it gets a bunch of numbers $x_1, \dots, x_n$ and can calculate a weighted sum $w_1 x_1 + \dots + w_n x_n$. What then should it do if you pass it a couple of lists $(x_1^1, \dots, x_1^{m_1}), \dots, (x_n^1, \dots, x_n^{m_n})$?

Edit 1: Let me be a bit clearer about this point. Every component of a neural network is built on the assumption that the input tensors contain real values. Mathematically: The input must be of the form $X \in \mathbb{R}^{n_1 \times \dots \times n_k}.$ What you would like instead are elements of the set containing all real vectors of arbitrary length, $\mathcal{R} := \{x \in \mathbb{R}^m \mid m \in \mathbb{N}\}$: You want to input $X \in \mathcal{R}^{n_1 \times \dots \times n_k}.$ This would require you to rethink and redefine absolutely everything: Activations, loss functions, forward propagation, back propagation... End of edit 1.

Second, it is unnecessary. Say your input consists of a list $(x_1^1, \dots, x_1^m)$ and two scalar inputs $x_2$ and $x_3$. Then you should simply pass Keras the $m+2$ scalar inputs $x_1^1, \dots, x_1^m, x_2, x_3$. See JahKnows' answer for a nice implementation.

Things get a bit more tricky if your input lists have different lengths ... so your first observation contains a list with five items, your second observation a list with seven items, etc. In that case you need to use padding and masking.

Edit 2: You say in the comments that you are unhappy with padding because some of your sequences are very long. Let me point out some alternatives:

  • Use batch padding with variable length. This is common for networks that take sentences as input. The test set is ordered by sentence length and split into batches. Each batch is padded individually with as few pads as possible.
  • Cut off sequences, either by dropping the oldest entries (for a time series), or the smallest, or the largest, or the most extreme, or the least extreme, or whatever makes sense for your data.
  • Feed your network with derived properties of the sequence, not the sequence itself. The more I think about it, the more this seems like the best approach for your problem. A distribution is fully described by its moments. By feeding enough moments (and the sequence length) into your network, you retain almost all information about the it. Notice also that moments remain unchanged for permutations of a sequence. Since you say that your sequences are not time series, this might be a very important property (and one that padding does not have!)
  • $\begingroup$ Thanks, your answer is the closest one by far and sequence padding has crossed my mind. I guess that might be the way to go with the solution I have in mind. $\endgroup$ – I_Play_With_Data Mar 6 '18 at 15:58
  • 1
    $\begingroup$ You don't sound too happy about padding. Why not? $\endgroup$ – Elias Strehle Mar 6 '18 at 16:16
  • $\begingroup$ because, in theory, the sequence of events that I have in mind can be infinite. So in order to make that approach work, I would have an incessantly long array with a lot of padding just to account for edge cases. $\endgroup$ – I_Play_With_Data Mar 6 '18 at 16:26
  • 1
    $\begingroup$ Perhaps you could edit some details about the use case into your original question. Right now I cannot imagine an application that would call for arbitrarily long input sequences. You are aware that you do not have to feed the entire sequence history into your LSTM at each step, right? $\endgroup$ – Elias Strehle Mar 6 '18 at 16:30
  • 1
    $\begingroup$ I tend to forget that ;-) I would be excited to see more details. You could also think about using summary statistics (length, mean, variance, maximum, ...) as input instead of the full sequence. $\endgroup$ – Elias Strehle Mar 6 '18 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.