Training stateful LSTM with different number of sequences

Question

I'm using a stateful LSTM for stock market analysis, and I have varying amounts of data for each stock, ranging from 20 years to just a few weeks (i.e. for newly listed stocks).

I use 3 years of data as a minimum for training as I want to create some state within the network. I set a year as my sequence length, so if I have 12 years of data then I will submit 4 batches with 3 sequences in each. Only after I've submitted all batches do I then reset the network state ready for the next stock.

But is there any issue training with differing number of sequences? E.g. if I train with a company that has 20 years of data then the network will build up much more state than a company that I only have 3 years of data.

Answer 1

In the circumstance of stock market prediction, I think that when the sequence length reaches a certain point, the network will learn to forget the opening price 6 months ago or the volume 3 years back. That data is no longer relevant to the cell state, as recent events are more indicative of stock price changes.
In terms of the LSTM method of storing both long and short-term memory, the LSTM will have the functionality to "forget" old and irrelevant data from earlier in the sequence, maintaining a stable and efficient cell state regardless of sequence length (of course, the sequence should be long enough for the LSTM to build up any cell state at all).
Hope this helps.

Answer 2

Training a stateful LSTM model on sequences of varying lengths can potentially lead to differences in the final state of the network, as you have observed. This can be caused by the fact that the network's internal state is updated at each timestep, and the number of timesteps that the network processes for each sequence will depend on the length of the sequence. As a result, the final state of the network may be different for sequences of different lengths, which can affect the performance of the model.

To mitigate this issue, you can try using a fixed-length sequence for all of your training data, by padding the shorter sequences with additional timesteps of zeros or some other placeholder value. This will ensure that all of the sequences have the same length, and the network will process the same number of timesteps for each sequence. This can help to ensure that the final state of the network is consistent across all of your training data, and can improve the performance of the model.

Another option is to reset the state of the network after each sequence, instead of only resetting it after all of the sequences in a batch have been processed. This will ensure that the network's state is reset at the end of each sequence, regardless of its length, and the network will start each new sequence with a consistent initial state. This can help to prevent differences in the final state of the network for sequences of different lengths, and can improve the performance of the model.

Ultimately, the best approach will depend on the specific characteristics of your dataset and your goals for the model. You may need to experiment with different techniques and configurations to find the best solution for your problem.