The gradient descent algorithm is, most simply, w'(i) = w(i)-r*dC/dw(i) where w(i) are the old weights, w'(i) are the new weights, C is the cost, r is the learning rate. I'm aware of the graphical justification for this.
For one weight, this is w' = w - r*dC/dw.
Second, we also have this equation deltaC ~= sum(dC/dw(i) * deltaw(i) ), which is just the definition of linearity of C near the point that its derivative is calculated. For one weight, this is the same as deltaC/deltaw ~= dC/dw, e.g., the definition of derivative.
Let there be only one weight, and let s = -deltaC. Then we have -s = dC/dw * (w'-w), where we've split up deltaw into the original and perturbed value. Then w'-w = s * (1/ (dC/dw)), and w' = w -s * (1/dC/dw). (Since we want to reduce the cost, we want deltaC to be <= 0, so s is >= 0, and looks like a normal positive learning rate.)
What I haven't been able to understand is why I get two different answers for what appears to be the same operation, updating the weights to lower the cost. In one case I use dC/dw, and in the other, I use 1/(dC/dw.) In both cases, r and s are small positive numbers.
What am I missing?