Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I’m currently building sequence models for forecasting, and have tried using RNNs, LSTMs, and GRUs.
Something unusual I noticed was the highly unstable loss curves, where the loss sometimes goes back to the loss level in the first few epochs. Interesting, the severity of this decreases from RNNs to LSTMs to GRUs.
Would anyone have an idea why this occurs?
For reference, here are the loss curves for the following models, across 500 epochs.
There can be some other factors that affect this, such as using simulated annealing (in a NN context) or other learning rate schedules. Are you using a specific LR schedule?
A schedule might be that the LR decreases by 50%, every time the validation loss of 5 epochs in a row does not decrease. This will help get closer and closer to a minimum of the loss. However, we know it is possible to get stuck within a local minimum, which may be far from optimal, so we can shake things up by increasing the LR once again, which will essentially throw the algorithm our of tyhe local minima and on its way to a new minima (at least that is the hope). This kind of schedule often produces loss curves like the ones you see.
Another alternative is simple that your batch sizes are quite small, and every once ina while, you get a batch that consists of example which your model really struggles with, so the loss for that batch (and so the epoch) would spike in comparison to other epochs.
A final idea - thinking more about your data - if it is time-series e.g. of stock prices or weather - there could be a regime change/shift. Meaning that the underlying function or system suddenly switches to a new pattern. Something like this could throw your model off the scent for a while, and so produce bumps in the loss curve.
A small point on terminology: LSTM and GRU architectures are themselves RNNs. A recurrent network is one in which connections do not only move forward in a network, but can also go sidewards across a layer or indeed backwards. So it is a more general term, whereas LSTM/GRU layers are specific examples of RNNs.
If you can say a little more about your three model architectures, perhaps it might be clearer which names to use - and maybe even better understand these loss curves :-)
nn.RNN, nn.LSTM, and nn.GRU from Pytorch. pytorch.org/docs/stable/nn.html
$\endgroup$
Aug 16, 2018 at 11:08