Training an RNN with examples of different lengths in Keras

Question

I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for training.

In the keras documentation, it says the input to an RNN layer must have shape (batch_size, timesteps, input_dim). This suggests that all the training examples have a fixed sequence length, namely timesteps.

But this is not especially typical, is it? I might want to have the RNN operate on sentences of varying lengths. When I train it on some corpus, I will feed it batches of sentences, all of different lengths.

I suppose the obvious thing to do would be to find the max length of any sequence in the training set and zero pad it. But then does that mean I can't make predictions at test time with input length greater than that?

This is a question about Keras's particular implementation, I suppose, but I'm also asking for what people typically do when faced with this kind of a problem in general.

Answer 1

This suggests that all the training examples have a fixed sequence length, namely timesteps.

That is not quite correct, since that dimension can be None, i.e. variable length. Within a single batch, you must have the same number of timesteps (this is typically where you see 0-padding and masking). But between batches there is no such restriction. During inference, you can have any length.

Example code that creates random time-length batches of training data.

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed
from keras.utils import to_categorical
import numpy as np

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(LSTM(8, return_sequences=True))
model.add(TimeDistributed(Dense(2, activation='sigmoid')))

print(model.summary(90))

model.compile(loss='categorical_crossentropy',
              optimizer='adam')

def train_generator():
    while True:
        sequence_length = np.random.randint(10, 100)
        x_train = np.random.random((1000, sequence_length, 5))
        # y_train will depend on past 5 timesteps of x
        y_train = x_train[:, :, 0]
        for i in range(1, 5):
            y_train[:, i:] += x_train[:, :-i, i]
        y_train = to_categorical(y_train > 2.5)
        yield x_train, y_train

model.fit_generator(train_generator(), steps_per_epoch=30, epochs=10, verbose=1)

And this is what it prints. Note the output shapes are (None, None, x) indicating variable batch size and variable timestep size.

__________________________________________________________________________________________
Layer (type)                            Output Shape                        Param #
==========================================================================================
lstm_1 (LSTM)                           (None, None, 32)                    4864
__________________________________________________________________________________________
lstm_2 (LSTM)                           (None, None, 8)                     1312
__________________________________________________________________________________________
time_distributed_1 (TimeDistributed)    (None, None, 2)                     18
==========================================================================================
Total params: 6,194
Trainable params: 6,194
Non-trainable params: 0
__________________________________________________________________________________________
Epoch 1/10
30/30 [==============================] - 6s 201ms/step - loss: 0.6913
Epoch 2/10
30/30 [==============================] - 4s 137ms/step - loss: 0.6738
...
Epoch 9/10
30/30 [==============================] - 4s 136ms/step - loss: 0.1643
Epoch 10/10
30/30 [==============================] - 4s 142ms/step - loss: 0.1441

Answer 2

@kbrose seems to have a better solution

I suppose the obvious thing to do would be to find the max length of any sequence in the training set and zero pad it.

This is usually a good solution. Maybe try max length of sequence + 100. Use whatever works best for your application.

But then does that mean I can't make predictions at test time with input length greater than that?

Not necessarily. The reason a fixed length is used in keras, is because it greatly improves performance by creating tensors of fixed shapes. But that's only for training. After training, you'll have learned the right weights for your task.

Let's assume, after training for hours, you realise your model's max length wasn't big/small enough and you now need to change the time steps, just extract the learned weights from the old model, build a new model with the new time steps and inject the learned weights into it.

You can probably do this using something like:

new_model.set_weights(old_model.get_weights())

I haven't tried it out myself. Please try it and post your results here for everyone's benefit. Here are some links: one two