Sliding window leads to overfitting in LSTM?

Question

Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs?

For a simplified example, assume that we have to predict the sequence of characters:

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Is it bad (or better?) if I keep training my LSTM with the following minibatches:

A B C D E F G H I J K L M N, backprop, erase the cell

B C D E F G H I J K L M N O, backprop, erase the cell

 .... and so on, shifting by 1 every time?

Previously, I always trained it as:

A B C D E F G H I J K L M N,  backprop, erase the cell

O P Q R S T U V W X Y Z,  backprop, erase the cell

---

Instead of shifting by one, would it be better to slide the window by 2 entries instead, etc? What would that mean (in terms of precision/overfitting)?

---

Also, if I were to do the sliding-window approach in a Feed-forward network, would it result in overfitting? I would assume yes, because the network is exposed to the same information regions for a very long time. For example, it is exposed to E F G H I J K for a long time.

---

Edit:

Please remember that cell state is erased between training batches, so the LSTM will have a "hammer to head" at these times. It's unable to remember what was before OPQRSTUVWXYZ. This means that the LSTM is unable to ever learn that "O" follows the "M".

So, I thought (thus my entire question), why not to give it intermediate (overlapping) batch in between...and in that case why not use multiple overlapping minibatches - to me this would provide a smoother training? Ultimately, that would mean a sliding window for an LSTM.

---

Some useful info I've found after answer was accepted:

from here

The first word of the English translation is probably highly correlated with the first word of the source sentence. But that means decoder has to consider information from 50 steps ago, and that information needs to be somehow encoded in the vector. Recurrent Neural Networks are known to have problems dealing with such long-range dependencies. In theory, architectures like LSTMs should be able to deal with this, but in practice long-range dependencies are still problematic.

For example, researchers have found that reversing the source sequence (feeding it backwards into the encoder) produces significantly better results because it shortens the path from the decoder to the relevant parts of the encoder. Similarly, feeding an input sequence twice also seems to help a network to better memorize things. For example, if one training example is "John went home", you would give "John went home John went home" to the network as one input.

Edit after accepting the answer:

Several months after, I am more inclined to use the sliding window approach, as it uses the data better. But in that case you probably don't want to train BCDEFGHIJKLMNO right after ABCDEFGHIJKLMN. Instead, shuffle your examples, to gradually and uniformly "brush-in" all of the information into your LSTM. Give it HIJKLMNOPQRSTU after ABCDEFGHIJKLMNO etc. That's directly related to Catastrophic Forgetting. As always, monitor the Validation and Test set closely, and stop as soon as you see their errors steadily increasing

Also, the "hammer to head" issue can be improved, by using Synthetic Gradients. See its benefit here: (linked answer discusses its benefit of long sequences) https://datascience.stackexchange.com/a/32425/43077

Answer 1

Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, here, for framing forecasting as a supervised learning problem.

If your data are not very rich, then you may find that any LSTM at all overfits. There are a lot of parameters in an LSTM (in fact, there are $4(mn + n^2 + n)$ parameters, where $m$ is the input length and $n$ is the output length, according to this post).

Since LSTMs do not require fixed size input, they can find the optimal lookback number on their own. However, if you've done a prior autoregressive analysis and decided that, for example, the current time step is most correlated with the 10th previous time step, and not correlated with the 11th or any time steps further in the past, then you could perhaps save yourself some training time by feeding in fixed-length sequences. However, that kind of defeats the purpose of an LSTM.

If your data are not rich enough for an LSTM, I would recommend trying something much simpler, like an autoregressive model, and working your way up.

EDIT (responding to a comment):

Overlapping sequences are used as input, especially when the sequence is very long (although, of course, 'long' is relative). Although LSTMs are better than a vanilla RNN for long sequences, they can still have some trouble remembering time steps from the beginning of a sequence if the sequence is very long. That led to things like the bidirectional LSTM, which reads the sequence forwards and backwards, improving the exposure of the network to the beginning and end of each input sequence. The principle is the same with overlapping sequences, although I would argue that overlapping sequences is more intuitive.

Answer 2

LSTMs do not require a sliding window of inputs. They can remember what they have seen in the past, and if you feed in training examples one at a time they will choose the right size window of inputs to remember on their own.

LSTM's are already prone to overfitting, and if you feed in lots of redundant data with a sliding window then yes, they are likely to overfit.

On the other hand, a sliding window is necessary for time series forecasting with Feedforward Neural Networks, because FNNs require a fixed size input and do not have memory, so this is the most natural way to feed them time series data.

Whether or not the FNN will overfit depends on its architecture and your data, but all standard regularization techniques will apply if it does. For example you can try choosing a smaller network, L2 regularization, Dropout, etc.