I am training a CNN (SSD Inception V2) and I get a strange shape of the training loss:

training loss overview

At first, I thought it was a too large learning rate (suggested in Question Oscillating loss in CNN ). But after taking a closer look, I don't think that's the case. When I zoom in to where the increase in loss happens, it looks like this:

loss detail

My interpretation is that this cannot be caused by learning rate because the individual steps made in the training are relatively small. But there is something the causes the training loss to start increasing at some point.

Question: What might be the cause of this? I am not looking for a technical answer or a fix but rather for an answer that would help me understand the behaviour on a conceptual level.

Thank you!

  • $\begingroup$ You don't really know how the optimization landscape for the parameters looks like. The learning rate could still be too big or the optimizer you are using is having difficulties. Have you tried a different learning rate or optimizer? $\endgroup$
    – bonfab
    Feb 4 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.