So in a classic gradient descent we have
W -= epsilon * (dL / dW)
However, how does this make sense if we consider that L and W have some units? Wouldn't it be more sensible to consider instead
W -= epsilon * (dW / dL)
(where epsilon now represents a small nudge in loss and has corresponding units)
Note also that if we are not on a point with 0 derivative we can consider a "local inverse" of the function L(W)
and we would have that dL/dW = 1/ (dW/dL)
. Therefore the signs of these two alternatives are equal - the only thing that differs is the size of the modification we apply to W.
In the standard way we take a larger step on a steeper region. "My" variant does the opposite. Imagine that a change of W results in a very small change of loss: then it would make sense to overcompensate for this during the update of W, in line with my proposed method. Where am I wrong?