2
$\begingroup$

I am still finding confusing on look back topic when using LSTM for time-series analysis. If I have hourly data and I want to predict next 6 hours with multiple predictors, should I look back up to 6 hours when I prepare my training set?

OR

Should I look back one hour (shift 1) and then predict next hour and take that predicted value and feed it back to predict the value after until next 6 hours?

This concept is still a little fuzzy for me in LSTM. Any thoughts would be appreciated.

$\endgroup$
  • 1
    $\begingroup$ What package are you using? Using Keras, you can certainly predict up to 6 hours (Looking back one hour, then feeding the predicted value is unnecessary work). How far you look back will likely need to be tuned as there is no rule of thumb. $\endgroup$ – Hobbes Sep 6 '17 at 17:11
  • $\begingroup$ @Hobbes I use keras with lstm. I could predict for next 6 hours looking back one hour. However, I have some predicted future values as my predictors and I tried MLP, it works great. As lstm can take the output with other inputs (predicted values of predictors), I was wondering if I should consider feeding predicted values. $\endgroup$ – i.n.n.m Sep 6 '17 at 21:18
  • $\begingroup$ @Hobbes to your point "Looking back one hour, then feeding the predicted value is unnecessary work", I gave lots of thought and I think MLP is the best option then. However, if were to consider an LSTM, that will be really challenging as we are not sure about the predicted values (which will be inputted with other inputs) might be off. $\endgroup$ – i.n.n.m Sep 17 '17 at 22:30
  • $\begingroup$ It is not unreasonable that MLP is performing better. I'm still a little confused with your exact method, but if you are getting good results, stick with it! $\endgroup$ – Hobbes Sep 18 '17 at 15:56
1
$\begingroup$

The second option sounds fun but better don't do it: errors will sum up and it's much harder to track where errors come from. Like you have some error for the 1st hour, feed it back in the net, predict the 2nd hour get an error there and can't really tell how much of that is caused by the error of the first hour.

Besides that, I haven't seen any application which works with this feedback mechanism.

| improve this answer | |
$\endgroup$
  • $\begingroup$ thank you for your thoughts. As you mentioned, 2nd option was more prone to errors. Predictions were bad. I have to make some optimal decisions i think! $\endgroup$ – i.n.n.m Oct 12 '17 at 14:07
1
$\begingroup$

So at time $t$ we want to predict values $y_{t+1}$ up to $y_{t+6}$ correct? You should use $x_0$ up to $x_t$ as inputs and use 6 values as your target/output. Then when you get new information, you add $x_{t+1}$ and use it to update your cell state and hidden state of your LSTM and get new outputs. The problem with feeding predictions is that errors will accumulate, although this does happen with sequence generation models (like seq2seq). What you would do then is during training use two types of training, one where you use current predictions and one where you use your known training data, both a fraction of the time. This is called 'teacher forcing'.

If we use my first suggestion, here is an explanation how to prepare your data. To train your model you will need to use lags to prepare your targets. Let's say we have $x$ as one-dimensional and we are trying to predict the next 6 values of $x$, you would get as $x$ and $y$:

| t | x   | y                     |
| 0 | 0   | [-1, 1, 2, 3, 2, 1]   |
| 1 | -1  | [1, 2, 3, 2, 1, 0]    |
| 2 | 1   | [2, 3, 2, 1, 0, -1]   |
| 3 | 2   | [3, 2, 1, 0, -1, -2]  |
| 4 | 3   | NaN                   |
| 5 | 2   | NaN                   |
| 6 | 1   | NaN                   |
| 7 | 0   | NaN                   |
| 8 | -1  | NaN                   |
| 9 | -2  | NaN                   |

The last six steps don't have a target because it's not available in our training set. The loss function would be a (weighted) mean squared error over the six predictions per time step. If you really don't want to throwaway data, you could make a custom loss function that can use the fact that only 4 targets are available because we are near the end, but that will complicate things.

EDIT: With regards to the question in the title, you would like to look back as far as possible, which basically means you always start at $t=0$.

| improve this answer | |
$\endgroup$
  • $\begingroup$ van der Vgt, this is neat. I didn't know about 'teacher forcing' before, I have to find some good reading on that. You are correct, feeding predictions were more towards accumulating error! $\endgroup$ – i.n.n.m Oct 12 '17 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.