6
$\begingroup$

Let's say I have a set of n time-series with sequence length 8

[[a,b,c,d,e,f,g,h],[f,e,g,r,g,h,e,a],[a,e,r,a,k,e,l,i],...,[e,r,q,g,l,r,p,q]]

And let's define the input that LSTM expects as a tensor of shape (samples,sequence-length,features)

I want to predict the last value of each one of them.

I cant either feed the network a sequence of the 7 first values with the 8th as the target:

[a,b,c,d,e,f,g] -> [h]

In this case the tensor would be (samples,7,1)

Or I can do the classic rolling window, with a window size of, say, 2.

[a,b], [b,c], [c,d], [d,e], [e,f], [f,g] -> [h]

In effect this shortens the length of the sequence.

And the input tensor would be (samples,2,1).

What are the trade-offs between performing rolling-windows or giving the "crude" time-series to the LSTM?

$\endgroup$
3
$\begingroup$

I cant either feed the network a sequence of the 7 first values with the 8th as the target:

For typical LSTM, if your input sequence is [a,b,c,d,e,f,g] then your target sequence will be [b, c,d,e,f, g, h]. You will train your network with a long sequence. And after training, you can simply pass in [h] as input to get the estimate for [i].

Or I can do the classic rolling window, with a window size of, say, 2.

There is no point in using LSTM if your window size is 2. You will end with one input and one output. There is no longer term dependency to be learned in this setting.

You might use a fixed window approach if your individual sequence is very long. You can slice your series using the window approach. The benefit of doing this

  • Reduce the length of the sequence. LSTM will still have problem learning dependency over very long steps due to gradient vanishing at the forget gate.
  • Augment the dataset in the similar way you doing rotation and scaling for an image dataset.

Summary

Say you have a single sequence [a,b,c,d,e,f,g,h,i]. Using a sliding window of size 6, your single squence will be transformed into 4 sequences

[
    [a,b,c,d,e,f],
    [b,c,d,e,f,g],
    [c,d,e,f,g,h],
    [d,e,f,g,h,i],
]

And you input X will

[
    [a,b,c,d,e],
    [b,c,d,e,f],
    [c,d,e,f,g],
    [d,e,f,g,h],
]

And your corresponding target Y will look like

[
    [b,c,d,e,f],
    [c,d,e,f,g],
    [d,e,f,g,h],
    [e,f,g,h,i],
]
| improve this answer | |
$\endgroup$
  • $\begingroup$ But why does my target need to be a sequence? Are there any advantages over framing it as a many-to-many instead of a many-to-one? rnn_image $\endgroup$ – Luciano Viola Nov 8 '17 at 3:51
  • 1
    $\begingroup$ This is just how RNN (including LSTM) works in general, gives an input it returns an output. So for a sequence of length n, it always corresponds to an output of length n. You can choose to ignore the outputs of the previous steps and only backpropagate using the output from the last step like what you described in the first case, it's just not how RNN is commonly used and you might not get a good result. That being said, this shouldn't stop you from trying. Highly recommend this post if you haven't read it already colah.github.io/posts/2015-08-Understanding-LSTMs $\endgroup$ – Louis T Nov 8 '17 at 4:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.