# Gradient descent around optimal loss surface

All the loss surface used in examples have some of bowl shape that decrease drastically far from the optimal and decrease slowly around the optimal flat point.

My questions are:

1. Has all the loss surface flat area around the optimal thus causing small gradient updates?

2. Most of loss curve when I trained some model shows some drastic decrease in loss and shows the decreasing become slow and the loss graph become flatten. Why the loss updates (amount of the gradient) become small ?

## 1 Answer

Based on how the gradient descent optimization algorithm works (which you can find extensevily explained in this link), find below the answers to your questions:

Has all the loss surface flat area around the optimal thus causing small gradient updates?

• as far as the updated weights approximate the optimal weight values (and this happens on what you call flat area of the loss function surface), the weights updates are smaller and smaller, and this is because the derivative of such loss function on each component becomes smaller (i.e. that nearly flat area is less steep):

and in terms of the gradient descent algorithm details below:

Most of loss curve when I trained some model shows some drastic decrease in loss and shows the decreasing become slow and the loss graph become flatten. Why the loss updates (amount of the gradient) become small?

• this second question (if I got it right you are asking about the learning curves) is a direct consequence of the point above: when the weights become near the optimal values when converging, the amount of weight update become smaller and the loss curve becomes nearly (with some oscillations) flat at around the minimum value, something like: source