2
$\begingroup$

I just found the animation below from Alec Radford's presentation:

enter image description here

As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of it. Regular SGD itself is simply stuck at the saddle point.

Why is this happening? Isn't the "movement speed" constant value that is dependent on the learning rate?

For example, weight for each point on regular SGD algorithm would be:

$$w_{t+1}=w_t-v*\frac{\partial L}{\partial w}$$

where $v$ is a learning rate and $L$ is a loss function.

In short, why are all optimization algorithms slowed down by the saddle point even though step size is constant value? Shouldn't a movement speed be constantly same?

$\endgroup$
2
$\begingroup$

In that simulation, the movement speed is a proxy for step size. The step size is a function of learning rate (v) and the approximate gradient of the function at that point ($\frac{\partial L}{\partial w}$). The learning rate can be constant. However, the approximate gradient is not constant. The approximate gradient is typically smaller closer to a critical point (i.e., the valley starts to gradually level out). Thus, the computed update value is smaller and the movement speed slows down.

$\endgroup$
  • $\begingroup$ My apologies for confusion. But what is meant by approximate gradient? Loss function that I've utilized is binary cross entropy loss, and it's partial derivative with respect to weight is $(y_{i} - p_{i})*x_{i}$ where $y_i$ is the actual value and $p_{i}$ is an estimated one - but isn't this derivative completely accurate? is there something else meant by gradient? Thank you! $\endgroup$ – ShellRox Aug 11 '18 at 20:19
  • $\begingroup$ It is the approximate gradient because SGD only uses a subset of data points to calculate the gradient. The calculation of the actual gradient would require a pass over all data points. $\endgroup$ – Brian Spiering Aug 11 '18 at 21:11
  • $\begingroup$ So the training is actually performed on the mini-batched dataset (that SGD randomly samples the full dataset and computes an average gradient). And during iteration average gradient changes but slows down when reaches a critical point. Thank you for the help! $\endgroup$ – ShellRox Aug 11 '18 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.