With stateful LSTM the entire state is retained between both the sequences in the batch that is submitted, and even between separate batches until
model.reset_states() is called. So my question is, does sequence length matter?
E.g. if my entire data set is 1000 rows, and each row has 5 attributes, and I set sequence length of 100 then my
[10,100,5], and the shape of my Y labels would be
[10,3] (3 Y labels, for example)
But as the state is retained between sequences, what difference does it make what length the sequence is as eventually the entire sequence is fed in and the state is retained?
Am I right in thinking that because I specify Y labels for each sequence (10 in this example), then that's the difference? i.e. it basically defines how many set of labels I can provide for an entire sequence.
For example, if I were to change sequence length to 200 then the batch shape would be
[5,200,5] and I'd therefore only provide 5 sets of Y labels for entire batch of sequences. That will change the network's learning behaviour quite substantially, correct?
So, essentially, do I treat it as a hyperparameter and experiment with what sequence length provides the best result?
It's funny with NNs; it seems you need a genetic algorithm to find the combination of hyperparameters that works best.