The gradient descent algorithm is, most simply, w'(i) = w(i)-r*dC/dw(i) where w(i) are the old weights, w'(i) are the new weights, C is the cost, r is the learning rate. I'm aware of the graphical justification for this.

For one weight, this is w' = w - r*dC/dw.

Second, we also have this equation deltaC ~= sum(dC/dw(i) * deltaw(i) ), which is just the definition of linearity of C near the point that its derivative is calculated. For one weight, this is the same as deltaC/deltaw ~= dC/dw, e.g., the definition of derivative.

Let there be only one weight, and let s = -deltaC. Then we have -s = dC/dw * (w'-w), where we've split up deltaw into the original and perturbed value. Then w'-w = s * (1/ (dC/dw)), and w' = w -s * (1/dC/dw). (Since we want to reduce the cost, we want deltaC to be <= 0, so s is >= 0, and looks like a normal positive learning rate.)

What I haven't been able to understand is why I get two different answers for what appears to be the same operation, updating the weights to lower the cost. In one case I use dC/dw, and in the other, I use 1/(dC/dw.) In both cases, r and s are small positive numbers.

What am I missing?


Your second function is not an update rule, it is just a re-statement of the approximation between using finite differences to measure a gradient and the infinitesimal ones used in calculus. Most importantly, $s$ is not a learning rate or parameter that you are free to change. It is, by definition, the (negative of the) change observed in $C$ when you alter $w$ to $w'$.

The second equation you have derived is the same as:

$$w + \Delta w \approx w + \Delta C \frac{dw}{dC}$$

In addition, for some reason you decided to drop the approximation symbol partway through and treat your manipulations as if you had created a new weight update rule. Be careful when manipulating expressions on either side of an approximation, they do not always follow the same rules as for an equality, and treating them as such can lead you down some false paths.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.