Is it necessary to perform rolling-window on LSTMs?

Question

Let's say I have a set of n time-series with sequence length 8

[[a,b,c,d,e,f,g,h],[f,e,g,r,g,h,e,a],[a,e,r,a,k,e,l,i],...,[e,r,q,g,l,r,p,q]]

And let's define the input that LSTM expects as a tensor of shape (samples,sequence-length,features)

I want to predict the last value of each one of them.

I cant either feed the network a sequence of the 7 first values with the 8th as the target:

[a,b,c,d,e,f,g] -> [h]

In this case the tensor would be (samples,7,1)

Or I can do the classic rolling window, with a window size of, say, 2.

[a,b], [b,c], [c,d], [d,e], [e,f], [f,g] -> [h]

In effect this shortens the length of the sequence.

And the input tensor would be (samples,2,1).

What are the trade-offs between performing rolling-windows or giving the "crude" time-series to the LSTM?

Answer 1

I cant either feed the network a sequence of the 7 first values with the 8th as the target:

For typical LSTM, if your input sequence is [a,b,c,d,e,f,g] then your target sequence will be [b, c,d,e,f, g, h]. You will train your network with a long sequence. And after training, you can simply pass in [h] as input to get the estimate for [i].

Or I can do the classic rolling window, with a window size of, say, 2.

There is no point in using LSTM if your window size is 2. You will end with one input and one output. There is no longer term dependency to be learned in this setting.

You might use a fixed window approach if your individual sequence is very long. You can slice your series using the window approach. The benefit of doing this

• Reduce the length of the sequence. LSTM will still have problem learning dependency over very long steps due to gradient vanishing at the forget gate.
• Augment the dataset in the similar way you doing rotation and scaling for an image dataset.

Summary

Say you have a single sequence [a,b,c,d,e,f,g,h,i]. Using a sliding window of size 6, your single squence will be transformed into 4 sequences

[
    [a,b,c,d,e,f],
    [b,c,d,e,f,g],
    [c,d,e,f,g,h],
    [d,e,f,g,h,i],
]

And you input X will

[
    [a,b,c,d,e],
    [b,c,d,e,f],
    [c,d,e,f,g],
    [d,e,f,g,h],
]

And your corresponding target Y will look like

[
    [b,c,d,e,f],
    [c,d,e,f,g],
    [d,e,f,g,h],
    [e,f,g,h,i],
]