I am training the following deeplab CNN: https://github.com/tensorflow/models/tree/master/research/deeplab
The first 50k steps of the training the loss is quite stable and low, and suddenly it starts to exponentially explode. I wonder how this can happen. Of course there are many reasons a loss can increase, such as a too high learning rate. But what I do not understand is the following:
- I use a batch size of 16 and I have 24k images, so 24k/16=1500 steps are used for a full pass on the train data
- Only after 50k steps the loss starts exploding, before that it is remarkably stable.
- So around the 34th iteration through my train set the loss starts to increase all of a sudden. Why only now? How can it be stable for so long and suddenly increase sharply?