A fundamental question about the gradient descent equation

Question

So in a classic gradient descent we have

W -= epsilon * (dL / dW)

However, how does this make sense if we consider that L and W have some units? Wouldn't it be more sensible to consider instead

W -= epsilon * (dW / dL) (where epsilon now represents a small nudge in loss and has corresponding units)

Note also that if we are not on a point with 0 derivative we can consider a "local inverse" of the function L(W) and we would have that dL/dW = 1/ (dW/dL). Therefore the signs of these two alternatives are equal - the only thing that differs is the size of the modification we apply to W.

In the standard way we take a larger step on a steeper region. "My" variant does the opposite. Imagine that a change of W results in a very small change of loss: then it would make sense to overcompensate for this during the update of W, in line with my proposed method. Where am I wrong?

Answer 1

You are looking for a local minimum, instead of a global minimum value.

The local minimum has $\frac{dL}{dW} = 0$, there is no direction of $W$ in which you can further reduce the loss.

If you are near the local minimum, you want to take small steps, not to 'overshoot'. If you are further away, you can take larger steps to speed up things, because you do have an idea about the direction of the minimum, namely the gradient.