What can be the cause of a sudden explosion in the loss when training a CNN (Deeplab)

Question

I am training the following deeplab CNN: https://github.com/tensorflow/models/tree/master/research/deeplab

During training I see the following loss:

The first 50k steps of the training the loss is quite stable and low, and suddenly it starts to exponentially explode. I wonder how this can happen. Of course there are many reasons a loss can increase, such as a too high learning rate. But what I do not understand is the following:

• I use a batch size of 16 and I have 24k images, so 24k/16=1500 steps are used for a full pass on the train data
• Only after 50k steps the loss starts exploding, before that it is remarkably stable.
• So around the 34th iteration through my train set the loss starts to increase all of a sudden. Why only now? How can it be stable for so long and suddenly increase sharply?

Answer 1

One possible reason could be the numerical instability of some weights or gradients.

For example, some weights or gradients might become too small, so when you do the calculations with them, it gives incorrect ("exploding") results. Same could happen if they become too large.

To make sure that this doesn't happen, it is recommended to do the clipping in some critical parts of the network. I would first try to do the clipping in the loss calculation. One example is here:

https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py#L3582

It's the cross-entropy calculated in Keras - this line clips the logit values before calculating the final loss. The reason for clipping is the tf.log calculation: the log(0) is problematic from the obvious reason (it's minus infinity) and the log(1) gives 0 which would produce no signal in the backpropagation.

Of course, this could be one of the reasons for the problem you have. From my experience, this would be the first thing I would check. I would help you if you inspect (print) the values of the numbers which could be suspects (e.g. values during the loss calculation, ...)

Also, make sure that the input data to your network is normalized (from [0, 1] interval). Otherwise, your weights would be very large and if you have large weights, just a slight change in the input could produce a very different output (in other words, the network becomes unstable, too sensitive to input changes).

Answer 2

One possible cause is a high learning rate. High values of this hyperparameter usually cause updates that are too drastic, and therefore divergence from the optimum.

Please keep in mind this is only a suggestion, your problem might be due to completely different reasons. Try different learning rates and schedules, in order to understand if that's the case. You can check this very good post on how to set the learning rate.

Also, consider using early stopping to save some training time.

Answer 3

I met a similar issue to yours but in a different task and with a different model architecture (GNNs). A potential answer to this phenomenon is the instability of the training process due to the flawed training data and the design of our models.

To overcome this issue, I tried several approaches.

First is clipping the gradients by calling clip_grad_value_ or clip_grad_norm_. However, it fails because this clipping only tackles training collapse when some outlier samples produce the gradient peak.

Secondly, I used weight decay to normalize the Adam optimizer. It also does not work for me because my model size is not very large and this sort of penalty has little effect on the training.

Thirdly, I utilized the warmup technique to prevent the fluctuation of the gradients at the beginning of the model training. The warmup is a mechanism introduced in ResNet. You set up a small learning rate at first and after the model parameters become more stable, you return to a large learning rate. This solved my problem.

You can make attempts at all three possible ways and see if they can help.