55
$\begingroup$

I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for training.

In the keras documentation, it says the input to an RNN layer must have shape (batch_size, timesteps, input_dim). This suggests that all the training examples have a fixed sequence length, namely timesteps.

But this is not especially typical, is it? I might want to have the RNN operate on sentences of varying lengths. When I train it on some corpus, I will feed it batches of sentences, all of different lengths.

I suppose the obvious thing to do would be to find the max length of any sequence in the training set and zero pad it. But then does that mean I can't make predictions at test time with input length greater than that?

This is a question about Keras's particular implementation, I suppose, but I'm also asking for what people typically do when faced with this kind of a problem in general.

$\endgroup$
  • $\begingroup$ @kbrose is correct. However, I have one concern. In the example, you have a very special generator of infinitely yields. More importantly, it is designed to yield batches of size 1000. In practice, this is too hard to satisfy, if not impossible. You need to re-organize your entries so those with the same length are arranged together, and you need to carefully set batch split positions. Moreover, you have no chance to make shuffle across the batches. So my opinion is: never use varying length input in Keras unless you exactly know what you are doing. Use padding and set Masking layer to ignor $\endgroup$ – Bs He Apr 16 at 23:10
49
$\begingroup$

This suggests that all the training examples have a fixed sequence length, namely timesteps.

That is not quite correct, since that dimension can be None, i.e. variable length. Within a single batch, you must have the same number of timesteps (this is typically where you see 0-padding and masking). But between batches there is no such restriction. During inference, you can have any length.

Example code that creates random time-length batches of training data.

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed
from keras.utils import to_categorical
import numpy as np

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(LSTM(8, return_sequences=True))
model.add(TimeDistributed(Dense(2, activation='sigmoid')))

print(model.summary(90))

model.compile(loss='categorical_crossentropy',
              optimizer='adam')

def train_generator():
    while True:
        sequence_length = np.random.randint(10, 100)
        x_train = np.random.random((1000, sequence_length, 5))
        # y_train will depend on past 5 timesteps of x
        y_train = x_train[:, :, 0]
        for i in range(1, 5):
            y_train[:, i:] += x_train[:, :-i, i]
        y_train = to_categorical(y_train > 2.5)
        yield x_train, y_train

model.fit_generator(train_generator(), steps_per_epoch=30, epochs=10, verbose=1)

And this is what it prints. Note the output shapes are (None, None, x) indicating variable batch size and variable timestep size.

__________________________________________________________________________________________
Layer (type)                            Output Shape                        Param #
==========================================================================================
lstm_1 (LSTM)                           (None, None, 32)                    4864
__________________________________________________________________________________________
lstm_2 (LSTM)                           (None, None, 8)                     1312
__________________________________________________________________________________________
time_distributed_1 (TimeDistributed)    (None, None, 2)                     18
==========================================================================================
Total params: 6,194
Trainable params: 6,194
Non-trainable params: 0
__________________________________________________________________________________________
Epoch 1/10
30/30 [==============================] - 6s 201ms/step - loss: 0.6913
Epoch 2/10
30/30 [==============================] - 4s 137ms/step - loss: 0.6738
...
Epoch 9/10
30/30 [==============================] - 4s 136ms/step - loss: 0.1643
Epoch 10/10
30/30 [==============================] - 4s 142ms/step - loss: 0.1441
$\endgroup$
  • $\begingroup$ Thank you for this. However, if we 0 pad the sequences, it will affect the hidden states and memory cell because we continue passing x_t as 0s, when if fact, there should be nothing passed. In the normal fit(), we can pass the sequence_lenth parameter to specify the length of the sequence to exclude it. It seems that the generator approach doesn't allow ignoring 0 sequences? $\endgroup$ – GRS Oct 1 '18 at 8:57
  • 1
    $\begingroup$ @GRS Your generator can return a 3-tuple of (inputs, targets, sample_weights), and you can set sample_weights of your 0-pads to 0. However, I'm not sure this would work perfectly for Bidirectional RNNs. $\endgroup$ – kbrose Oct 1 '18 at 14:28
  • $\begingroup$ This has been helpful, but I wish it would also include an example of using model.predict_generator with a test set. When I try to predict with a generator I get an error regarding concatenation (the test set also has variable length sequences). My solution has been to use the standard model.predict in a hacky way. Perhaps this would just be better fit for a new question? $\endgroup$ – mickey Jan 22 at 16:55
  • $\begingroup$ @mickey that sounds like a different question. This question is about training, not prediction. $\endgroup$ – kbrose Jan 29 at 13:35
6
$\begingroup$

@kbrose seems to have a better solution

I suppose the obvious thing to do would be to find the max length of any sequence in the training set and zero pad it.

This is usually a good solution. Maybe try max length of sequence + 100. Use whatever works best for your application.

But then does that mean I can't make predictions at test time with input length greater than that?

Not necessarily. The reason a fixed length is used in keras, is because it greatly improves performance by creating tensors of fixed shapes. But that's only for training. After training, you'll have learned the right weights for your task.

Let's assume, after training for hours, you realise your model's max length wasn't big/small enough and you now need to change the time steps, just extract the learned weights from the old model, build a new model with the new time steps and inject the learned weights into it.

You can probably do this using something like:

new_model.set_weights(old_model.get_weights())

I haven't tried it out myself. Please try it and post your results here for everyone's benefit. Here are some links: one two

$\endgroup$
  • 1
    $\begingroup$ You can indeed have variable length inputs, no need to introduce hacks like max length + 100. See my answer for example code. $\endgroup$ – kbrose Feb 16 '18 at 18:50
  • 1
    $\begingroup$ Transferring the weights to a model with more timesteps does indeed work perfectly fine! I bumped up the timesteps for Bidirectional(LSTM)() and RepeatVector() layers, and the predictions are perfectly viable. $\endgroup$ – komodovaran_ Mar 25 at 14:50
  • $\begingroup$ @kbrose This is not a hack, it is how you normally do it. Using a batch_size of one is too slow and keras enable masking layers so that the masking doesn't affect loss. $\endgroup$ – Ferus Aug 21 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.