So in a classic gradient descent we have

W -= epsilon * (dL / dW)

However, how does this make sense if we consider that L and W have some units? Wouldn't it be more sensible to consider instead

W -= epsilon * (dW / dL) (where epsilon now represents a small nudge in loss and has corresponding units)

Note also that if we are not on a point with 0 derivative we can consider a "local inverse" of the function L(W) and we would have that dL/dW = 1/ (dW/dL). Therefore the signs of these two alternatives are equal - the only thing that differs is the size of the modification we apply to W.

In the standard way we take a larger step on a steeper region. "My" variant does the opposite. Imagine that a change of W results in a very small change of loss: then it would make sense to overcompensate for this during the update of W, in line with my proposed method. Where am I wrong?

The local minimum has $$\frac{dL}{dW} = 0$$, there is no direction of $$W$$ in which you can further reduce the loss.