# Where is my error in understanding gradient descent calculated two different ways?

The gradient descent algorithm is, most simply, w'(i) = w(i)-r*dC/dw(i) where w(i) are the old weights, w'(i) are the new weights, C is the cost, r is the learning rate. I'm aware of the graphical justification for this.

For one weight, this is w' = w - r*dC/dw.

Second, we also have this equation deltaC ~= sum(dC/dw(i) * deltaw(i) ), which is just the definition of linearity of C near the point that its derivative is calculated. For one weight, this is the same as deltaC/deltaw ~= dC/dw, e.g., the definition of derivative.

Let there be only one weight, and let s = -deltaC. Then we have -s = dC/dw * (w'-w), where we've split up deltaw into the original and perturbed value. Then w'-w = s * (1/ (dC/dw)), and w' = w -s * (1/dC/dw). (Since we want to reduce the cost, we want deltaC to be <= 0, so s is >= 0, and looks like a normal positive learning rate.)

What I haven't been able to understand is why I get two different answers for what appears to be the same operation, updating the weights to lower the cost. In one case I use dC/dw, and in the other, I use 1/(dC/dw.) In both cases, r and s are small positive numbers.

What am I missing?

Your second function is not an update rule, it is just a re-statement of the approximation between using finite differences to measure a gradient and the infinitesimal ones used in calculus. Most importantly, $$s$$ is not a learning rate or parameter that you are free to change. It is, by definition, the (negative of the) change observed in $$C$$ when you alter $$w$$ to $$w'$$.
$$w + \Delta w \approx w + \Delta C \frac{dw}{dC}$$