Using LSTMs for continous learning and predicting

Question

I'm trying to develop a model to predict a commodity price movement direction based on previous observations. The model should learn common technical analysis patterns, e.g. head and shoulders. So, I think I should use a stateful LSTM so that it maintain a long term state to keep track of technical analysis patterns.

On the other side, as the data set is updated daily, i.e. new observations are added, I need the model to keep learning and making predictions every day. So, in order to update the model parameters on each new observation, I think I should use batch_size=1.

If I use the last N observations to predict the next M steps, the input tensor to the model would have the shape of (1, N, num_features):

$$ X_1, X_2, ..., X_N \rightarrow Y_1, Y_2, ..., Y_M $$

Questions:

• Why should we reset_states after training on train set and prior to predication on test set? I think we shouldn't as a TA pattern may be half in the train set and half in the test set. If we reset the states, the model can't recognize this pattern.

• Can I reset_states and feed in data of a different commodity? as I only need the model to learn TA patterns not commodity specific characteristics. If so, how can I restore the states to make prediction on the first commodity?

Answer 1

To answer your questions in order:

1. The whole point of LSTM (or any time series model) is to predict previously unseen values. If states are not reset - then there is a risk of data leakage - whereby forecasts from the training set will lead into the test set. This would mean that your model would perform quite well on the existing set of data - but would perform badly on new/unseen data. This defeats the purpose of implementing a time series model.

2. Depending on how similar the pattern is, it might be an option. However, it is just my personal experience that LSTMs are not particularly effective at forecasting commodity prices - the main reason being that these models tend to carry forward too much volatility in the forecasts, which tends to interfere with the overall trend of the data. Here is an example of the use of LSTMs in forecasting oil prices, which you might find of interest: Are LSTMs Always Effective At Time Series Analysis? A Case Study of Oil Prices