0
$\begingroup$

I am training the following deeplab CNN: https://github.com/tensorflow/models/tree/master/research/deeplab

During training I see the following loss: enter image description here

The first 50k steps of the training the loss is quite stable and low, and suddenly it starts to exponentially explode. I wonder how this can happen. Of course there are many reasons a loss can increase, such as a too high learning rate. But what I do not understand is the following:

  • I use a batch size of 16 and I have 24k images, so 24k/16=1500 steps are used for a full pass on the train data
  • Only after 50k steps the loss starts exploding, before that it is remarkably stable.
  • So around the 34th iteration through my train set the loss starts to increase all of a sudden. Why only now? How can it be stable for so long and suddenly increase sharply?
$\endgroup$
0
$\begingroup$

One possible cause is a high learning rate. High values of this hyperparameter usually cause updates that are too drastic, and therefore divergence from the optimum.

Please keep in mind this is only a suggestion, your problem might be due to completely different reasons. Try different learning rates and schedules, in order to understand if that's the case. You can check this very good post on how to set the learning rate.

Also, consider using early stopping to save some training time.

$\endgroup$
  • $\begingroup$ Wouldn't too large learning rate produce the bad results (e.g. large loss) immediately from the training start? $\endgroup$ – Antonio Jurić Sep 6 at 7:17
  • $\begingroup$ Actually no, as far as I understood it happens when the gradient passes the optimum, and starts overshooting in the opposite direction (upwards). This is what I got from courses and books. Also, from my personal experience in training ANNs. Check also the first image of the link I sent you. $\endgroup$ – Leevo Sep 6 at 7:52
  • $\begingroup$ At the beginning of training, a large learning rate would actually make your gradient converge faster. It's around the last pit of the loss function that your model would benefit from smaller steps... and that's where too large gradient updates cause problems instead. $\endgroup$ – Leevo Sep 6 at 7:57
0
$\begingroup$

One possible reason could be the numerical instability of some weights or gradients.

For example, some weights or gradients might become too small, so when you do the calculations with them, it gives incorrect ("exploding") results. Same could happen if they become too large.

To make sure that this doesn't happen, it is recommended to do the clipping in some critical parts of the network. I would first try to do the clipping in the loss calculation. One example is here:

https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py#L3582

It's the cross-entropy calculated in Keras - this line clips the logit values before calculating the final loss. The reason for clipping is the tf.log calculation: the log(0) is problematic from the obvious reason (it's minus infinity) and the log(1) gives 0 which would produce no signal in the backpropagation.

Of course, this could be one of the reasons for the problem you have. From my experience, this would be the first thing I would check. I would help you if you inspect (print) the values of the numbers which could be suspects (e.g. values during the loss calculation, ...)

Also, make sure that the input data to your network is normalized (from [0, 1] interval). Otherwise, your weights would be very large and if you have large weights, just a slight change in the input could produce a very different output (in other words, the network becomes unstable, too sensitive to input changes).

$\endgroup$
  • $\begingroup$ Thanks for your hints, I will try them out as soon as possible and report back. $\endgroup$ – MuadDev Sep 6 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.