With stateful LSTM the entire state is retained between both the sequences in the batch that is submitted, and even between separate batches until model.reset_states() is called. So my question is, does sequence length matter?

E.g. if my entire data set is 1000 rows, and each row has 5 attributes, and I set sequence length of 100 then my batch_input_shape is [10,100,5], and the shape of my Y labels would be [10,3] (3 Y labels, for example)

But as the state is retained between sequences, what difference does it make what length the sequence is as eventually the entire sequence is fed in and the state is retained?

Am I right in thinking that because I specify Y labels for each sequence (10 in this example), then that's the difference? i.e. it basically defines how many set of labels I can provide for an entire sequence.

For example, if I were to change sequence length to 200 then the batch shape would be [5,200,5] and I'd therefore only provide 5 sets of Y labels for entire batch of sequences. That will change the network's learning behaviour quite substantially, correct?

So, essentially, do I treat it as a hyperparameter and experiment with what sequence length provides the best result?

It's funny with NNs; it seems you need a genetic algorithm to find the combination of hyperparameters that works best.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.