All the loss surface used in examples have some of bowl shape that decrease drastically far from the optimal and decrease slowly around the optimal flat point.
My questions are:
Has all the loss surface flat area around the optimal thus causing small gradient updates?
Most of loss curve when I trained some model shows some drastic decrease in loss and shows the decreasing become slow and the loss graph become flatten. Why the loss updates (amount of the gradient) become small ?