I’m currently building sequence models for forecasting, and have tried using RNNs, LSTMs, and GRUs.

Something unusual I noticed was the highly unstable loss curves, where the loss sometimes goes back to the loss level in the first few epochs. Interesting, the severity of this decreases from RNNs to LSTMs to GRUs.

Would anyone have an idea why this occurs?

For reference, here are the loss curves for the following models, across 500 epochs.





1 Answer 1


There can be some other factors that affect this, such as using simulated annealing (in a NN context) or other learning rate schedules. Are you using a specific LR schedule?

A schedule might be that the LR decreases by 50%, every time the validation loss of 5 epochs in a row does not decrease. This will help get closer and closer to a minimum of the loss. However, we know it is possible to get stuck within a local minimum, which may be far from optimal, so we can shake things up by increasing the LR once again, which will essentially throw the algorithm our of tyhe local minima and on its way to a new minima (at least that is the hope). This kind of schedule often produces loss curves like the ones you see.

Another alternative is simple that your batch sizes are quite small, and every once ina while, you get a batch that consists of example which your model really struggles with, so the loss for that batch (and so the epoch) would spike in comparison to other epochs.

A final idea - thinking more about your data - if it is time-series e.g. of stock prices or weather - there could be a regime change/shift. Meaning that the underlying function or system suddenly switches to a new pattern. Something like this could throw your model off the scent for a while, and so produce bumps in the loss curve.

A small point on terminology: LSTM and GRU architectures are themselves RNNs. A recurrent network is one in which connections do not only move forward in a network, but can also go sidewards across a layer or indeed backwards. So it is a more general term, whereas LSTM/GRU layers are specific examples of RNNs.

If you can say a little more about your three model architectures, perhaps it might be clearer which names to use - and maybe even better understand these loss curves :-)

  • $\begingroup$ In this case, I'm using very basic parameters (i.e., SGD with constant learning rate of 0.0025, batch size = 32, hidden layers = 1, hidden size = 64). On the final idea, each epoch goes through all the sequences of data, so it's unusual that the loss curves jump so much from epoch to epoch. On the architectures, they're really plain vanilla nn.RNN, nn.LSTM, and nn.GRU from Pytorch. pytorch.org/docs/stable/nn.html $\endgroup$
    – Eugene Yan
    Aug 16, 2018 at 11:08
  • $\begingroup$ @EugeneYan - Overall it seems as though the GRU architecture is best (lowest loss and smoothest curve). Try using a LR-scheduler to improve results. As for the spikes in the loss curves, I don't have any other idea right now! The only time I have really had that myself, was when I introduced new data to the trainingset during training, which of course explains the curve. Is anything else changing in your setup during training? $\endgroup$
    – n1k31t4
    Aug 16, 2018 at 11:17
  • $\begingroup$ No, nothing is changing. It's essentially a set of sequences with a binary label. Each epoch goes through all of the sequences in the set, so it's unusual it jumps around from set to set, though we're able to control it via the GRU somewhat. $\endgroup$
    – Eugene Yan
    Aug 16, 2018 at 12:31
  • $\begingroup$ In that case I can only think it is a certain order/set of samples that are randomly used within an epoch, which temporarily displaces the model's position in the loss-space to outside of the (local) minimum in which it more often resides. $\endgroup$
    – n1k31t4
    Aug 21, 2018 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.