0
$\begingroup$

All the loss surface used in examples have some of bowl shape that decrease drastically far from the optimal and decrease slowly around the optimal flat point.

My questions are:

  1. Has all the loss surface flat area around the optimal thus causing small gradient updates?

  2. Most of loss curve when I trained some model shows some drastic decrease in loss and shows the decreasing become slow and the loss graph become flatten. Why the loss updates (amount of the gradient) become small ?

$\endgroup$
0
$\begingroup$

Based on how the gradient descent optimization algorithm works (which you can find extensevily explained in this link), find below the answers to your questions:

Has all the loss surface flat area around the optimal thus causing small gradient updates?

  • as far as the updated weights approximate the optimal weight values (and this happens on what you call flat area of the loss function surface), the weights updates are smaller and smaller, and this is because the derivative of such loss function on each component becomes smaller (i.e. that nearly flat area is less steep):

gradient descent source

and in terms of the gradient descent algorithm details below:

enter image description here source

Most of loss curve when I trained some model shows some drastic decrease in loss and shows the decreasing become slow and the loss graph become flatten. Why the loss updates (amount of the gradient) become small?

  • this second question (if I got it right you are asking about the learning curves) is a direct consequence of the point above: when the weights become near the optimal values when converging, the amount of weight update become smaller and the loss curve becomes nearly (with some oscillations) flat at around the minimum value, something like: learning curve source
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.