My learning curve is behaving strangely and I don't know if I'm doing anything wrong or it is because of the dataset nature. I'm using a Neural network with (30,30,15,1) layers and 'ReLU' activation function for training and linear for the output layer. My learning curve is showing weird behavior while learning. There is a bump like increase and decrease in the validation curve around 100 epochs which I don't have any idea about the source or how to resolve it(or even if it is needed to be resolved). Can anybody help me with this? (Any related comments are welcomed) enter image description here


That pattern is common in neural networks training.

Train performance is an estimate of bias, and validation performance is an estimate of variance. Initially both go down. Bias continues to go down, but variance goes up. That is the classic bias-variance tradeoff. However, in neural networks variance will start to go down again. This is called the “double descent curve”.

It is not known why neural networks training show this “double descent curve.” One idea is that this pattern might be a concatenation of the two curves. One curve might be from optimization and the other curve might be from sampling. The paper "A Modern Take on the Bias-Variance Tradeoff" goes into more detail about it.

  • $\begingroup$ +1 Brian for yet another amazing answer! $\endgroup$ – Aditya Jan 29 '20 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.