# What can be the cause of a sudden explosion in the loss when training a CNN (Deeplab)

I am training the following deeplab CNN: https://github.com/tensorflow/models/tree/master/research/deeplab

During training I see the following loss:

The first 50k steps of the training the loss is quite stable and low, and suddenly it starts to exponentially explode. I wonder how this can happen. Of course there are many reasons a loss can increase, such as a too high learning rate. But what I do not understand is the following:

• I use a batch size of 16 and I have 24k images, so 24k/16=1500 steps are used for a full pass on the train data
• Only after 50k steps the loss starts exploding, before that it is remarkably stable.
• So around the 34th iteration through my train set the loss starts to increase all of a sudden. Why only now? How can it be stable for so long and suddenly increase sharply?

One possible reason could be the numerical instability of some weights or gradients.

For example, some weights or gradients might become too small, so when you do the calculations with them, it gives incorrect ("exploding") results. Same could happen if they become too large.

To make sure that this doesn't happen, it is recommended to do the clipping in some critical parts of the network. I would first try to do the clipping in the loss calculation. One example is here:

It's the cross-entropy calculated in Keras - this line clips the logit values before calculating the final loss. The reason for clipping is the tf.log calculation: the log(0) is problematic from the obvious reason (it's minus infinity) and the log(1) gives 0 which would produce no signal in the backpropagation.

Of course, this could be one of the reasons for the problem you have. From my experience, this would be the first thing I would check. I would help you if you inspect (print) the values of the numbers which could be suspects (e.g. values during the loss calculation, ...)

Also, make sure that the input data to your network is normalized (from [0, 1] interval). Otherwise, your weights would be very large and if you have large weights, just a slight change in the input could produce a very different output (in other words, the network becomes unstable, too sensitive to input changes).

• Thanks for your hints, I will try them out as soon as possible and report back. Sep 6, 2019 at 13:54