15
$\begingroup$

Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs?

For a simplified example, assume that we have to predict the sequence of characters:

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Is it bad (or better?) if I keep training my LSTM with the following minibatches:

A B C D E F G H I J K L M N, backprop, erase the cell

B C D E F G H I J K L M N O, backprop, erase the cell

 .... and so on, shifting by 1 every time?

Previously, I always trained it as:

A B C D E F G H I J K L M N,  backprop, erase the cell

O P Q R S T U V W X Y Z,  backprop, erase the cell

Instead of shifting by one, would it be better to slide the window by 2 entries instead, etc? What would that mean (in terms of precision/overfitting)?


Also, if I were to do the sliding-window approach in a Feed-forward network, would it result in overfitting? I would assume yes, because the network is exposed to the same information regions for a very long time. For example, it is exposed to E F G H I J K for a long time.


Edit:

Please remember that cell state is erased between training batches, so the LSTM will have a "hammer to head" at these times. It's unable to remember what was before OPQRSTUVWXYZ. This means that the LSTM is unable to ever learn that "O" follows the "M".

So, I thought (thus my entire question), why not to give it intermediate (overlapping) batch in between...and in that case why not use multiple overlapping minibatches - to me this would provide a smoother training? Ultimately, that would mean a sliding window for an LSTM.


Some useful info I've found after answer was accepted:

from here

The first word of the English translation is probably highly correlated with the first word of the source sentence. But that means decoder has to consider information from 50 steps ago, and that information needs to be somehow encoded in the vector. Recurrent Neural Networks are known to have problems dealing with such long-range dependencies. In theory, architectures like LSTMs should be able to deal with this, but in practice long-range dependencies are still problematic.

For example, researchers have found that reversing the source sequence (feeding it backwards into the encoder) produces significantly better results because it shortens the path from the decoder to the relevant parts of the encoder. Similarly, feeding an input sequence twice also seems to help a network to better memorize things. For example, if one training example is "John went home", you would give "John went home John went home" to the network as one input.

Edit after accepting the answer:

Several months after, I am more inclined to use the sliding window approach, as it uses the data better. But in that case you probably don't want to train BCDEFGHIJKLMNO right after ABCDEFGHIJKLMNO. Instead, shuffle your examples, to gradually and uniformly "brush-in" all of the information into your LSTM. Give it HIJKLMNOPQRSTU after ABCDEFGHIJKLMNO etc. That's directly related to Catastrophic Forgetting. As always, monitor the Validation and Test set closely, and stop as soon as you see their errors steadily increasing

Also, the "hammer to head" issue can be improved, by using Synthetic Gradients. See its benefit here: (linked answer discusses its benefit of long sequences) https://datascience.stackexchange.com/a/32425/43077

$\endgroup$
9
+50
$\begingroup$

Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, here, for framing forecasting as a supervised learning problem.

If your data are not very rich, then you may find that any LSTM at all overfits. There are a lot of parameters in an LSTM (in fact, there are $4(mn + n^2 + n)$ parameters, where $m$ is the input length and $n$ is the output length, according to this post).

Since LSTMs do not require fixed size input, they can find the optimal lookback number on their own. However, if you've done a prior autoregressive analysis and decided that, for example, the current time step is most correlated with the 10th previous time step, and not correlated with the 11th or any time steps further in the past, then you could perhaps save yourself some training time by feeding in fixed-length sequences. However, that kind of defeats the purpose of an LSTM.

If your data are not rich enough for an LSTM, I would recommend trying something much simpler, like an autoregressive model, and working your way up.

EDIT (responding to a comment):

Overlapping sequences are used as input, especially when the sequence is very long (although, of course, 'long' is relative). Although LSTMs are better than a vanilla RNN for long sequences, they can still have some trouble remembering time steps from the beginning of a sequence if the sequence is very long. That led to things like the bidirectional LSTM, which reads the sequence forwards and backwards, improving the exposure of the network to the beginning and end of each input sequence. The principle is the same with overlapping sequences, although I would argue that overlapping sequences is more intuitive.

$\endgroup$
  • $\begingroup$ Thank you, I I still don't see if LSTM should /shouldn't be trained with non-overlapping batches. The post you've liked is a valuable one, but it only discusses Feed-Forward nets and doesn't address benefits / dangers in LSTM's overlapping minibatches during training. @Imran also didn't discuss the "anti-prescription" against overlapping minibatches - my first comment to his post. $\endgroup$ – Kari Feb 14 '18 at 17:05
  • $\begingroup$ I've edited my question to include the comment $\endgroup$ – Kari Feb 14 '18 at 17:10
  • $\begingroup$ Lots of great info! $\endgroup$ – Imran Feb 14 '18 at 17:27
  • $\begingroup$ @Kari I've modified my answer. Does that help? $\endgroup$ – StatsSorceress Feb 14 '18 at 19:41
5
$\begingroup$

LSTMs do not require a sliding window of inputs. They can remember what they have seen in the past, and if you feed in training examples one at a time they will choose the right size window of inputs to remember on their own.

LSTM's are already prone to overfitting, and if you feed in lots of redundant data with a sliding window then yes, they are likely to overfit.

On the other hand, a sliding window is necessary for time series forecasting with Feedforward Neural Networks, because FNNs require a fixed size input and do not have memory, so this is the most natural way to feed them time series data.

Whether or not the FNN will overfit depends on its architecture and your data, but all standard regularization techniques will apply if it does. For example you can try choosing a smaller network, L2 regularization, Dropout, etc.

$\endgroup$
  • $\begingroup$ Thanks! I would argue that cell state is erased between training batches, so LSTM will have a "hammer to head" at these times. It's unable to remember what was before OPQRSTUVWXYZ. This means LSTM is unable to ever learn that "O" follows the "M". So I thought, why not to give it intermediate (overlapping) batch in between ...and in that case why not use multiple overlapping minibatches - to me this would provide a smoother training? $\endgroup$ – Kari Feb 12 '18 at 10:31
  • $\begingroup$ Ultimately, that would mean a sliding window for an LSTM $\endgroup$ – Kari Feb 12 '18 at 12:42
  • $\begingroup$ It's not necessary to erase the cell state in between training batches, though backpropagation further back is not possible of course. $\endgroup$ – Jan van der Vegt Feb 14 '18 at 16:17
  • $\begingroup$ I did try it, and - even with 0 learning rate the error was jumping up and down by tiny amount because incorrect cell states were re-used as "legacy" during training. I couldn't stack more than 6 layers with this approach - it got too chaotic and wouldn't converge. However, resetting cell state to zero after each Backprop allowed me to stack 150 layers with 64 neurons in each layer and train it with 0.001 learning rate & momentum of 0.9 (I am using LayerNormalization, that's why learning rate is so large in my case) $\endgroup$ – Kari Feb 14 '18 at 16:51
  • 1
    $\begingroup$ This means that with "legacy cellstates" LSTM becomes unstable & unreliable - it starts working on a new minibatch basing its decisions on the last cell-state (of previous minibatch) that wasn't corrected to the full extent. So, erasing the cell-state removes this fundimental flaw, but makes LSTM experience amnesia $\endgroup$ – Kari Feb 14 '18 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.