7
$\begingroup$

I am training the following deeplab CNN: https://github.com/tensorflow/models/tree/master/research/deeplab

During training I see the following loss: enter image description here

The first 50k steps of the training the loss is quite stable and low, and suddenly it starts to exponentially explode. I wonder how this can happen. Of course there are many reasons a loss can increase, such as a too high learning rate. But what I do not understand is the following:

  • I use a batch size of 16 and I have 24k images, so 24k/16=1500 steps are used for a full pass on the train data
  • Only after 50k steps the loss starts exploding, before that it is remarkably stable.
  • So around the 34th iteration through my train set the loss starts to increase all of a sudden. Why only now? How can it be stable for so long and suddenly increase sharply?
$\endgroup$
1

3 Answers 3

3
$\begingroup$

One possible reason could be the numerical instability of some weights or gradients.

For example, some weights or gradients might become too small, so when you do the calculations with them, it gives incorrect ("exploding") results. Same could happen if they become too large.

To make sure that this doesn't happen, it is recommended to do the clipping in some critical parts of the network. I would first try to do the clipping in the loss calculation. One example is here:

https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py#L3582

It's the cross-entropy calculated in Keras - this line clips the logit values before calculating the final loss. The reason for clipping is the tf.log calculation: the log(0) is problematic from the obvious reason (it's minus infinity) and the log(1) gives 0 which would produce no signal in the backpropagation.

Of course, this could be one of the reasons for the problem you have. From my experience, this would be the first thing I would check. I would help you if you inspect (print) the values of the numbers which could be suspects (e.g. values during the loss calculation, ...)

Also, make sure that the input data to your network is normalized (from [0, 1] interval). Otherwise, your weights would be very large and if you have large weights, just a slight change in the input could produce a very different output (in other words, the network becomes unstable, too sensitive to input changes).

$\endgroup$
3
  • $\begingroup$ Thanks for your hints, I will try them out as soon as possible and report back. $\endgroup$
    – MuadDev
    Sep 6, 2019 at 13:54
  • 2
    $\begingroup$ this link is now dead $\endgroup$ Jul 9, 2021 at 15:40
  • $\begingroup$ "and the log(1) gives 0 which would produce no signal in the backpropagation." I do not think this is correct, log(1)=0 still has a gradient. But otherwise a good explanation! $\endgroup$
    – markemus
    Dec 7, 2022 at 9:07
1
$\begingroup$

One possible cause is a high learning rate. High values of this hyperparameter usually cause updates that are too drastic, and therefore divergence from the optimum.

Please keep in mind this is only a suggestion, your problem might be due to completely different reasons. Try different learning rates and schedules, in order to understand if that's the case. You can check this very good post on how to set the learning rate.

Also, consider using early stopping to save some training time.

$\endgroup$
3
  • $\begingroup$ Wouldn't too large learning rate produce the bad results (e.g. large loss) immediately from the training start? $\endgroup$ Sep 6, 2019 at 7:17
  • $\begingroup$ Actually no, as far as I understood it happens when the gradient passes the optimum, and starts overshooting in the opposite direction (upwards). This is what I got from courses and books. Also, from my personal experience in training ANNs. Check also the first image of the link I sent you. $\endgroup$
    – Leevo
    Sep 6, 2019 at 7:52
  • $\begingroup$ At the beginning of training, a large learning rate would actually make your gradient converge faster. It's around the last pit of the loss function that your model would benefit from smaller steps... and that's where too large gradient updates cause problems instead. $\endgroup$
    – Leevo
    Sep 6, 2019 at 7:57
0
$\begingroup$

I met a similar issue to yours but in a different task and with a different model architecture (GNNs). A potential answer to this phenomenon is the instability of the training process due to the flawed training data and the design of our models.

To overcome this issue, I tried several approaches.

First is clipping the gradients by calling clip_grad_value_ or clip_grad_norm_. However, it fails because this clipping only tackles training collapse when some outlier samples produce the gradient peak.

Secondly, I used weight decay to normalize the Adam optimizer. It also does not work for me because my model size is not very large and this sort of penalty has little effect on the training.

Thirdly, I utilized the warmup technique to prevent the fluctuation of the gradients at the beginning of the model training. The warmup is a mechanism introduced in ResNet. You set up a small learning rate at first and after the model parameters become more stable, you return to a large learning rate. This solved my problem.

You can make attempts at all three possible ways and see if they can help.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.