0
$\begingroup$

I have been going over some theory for gradient descent. The source I am looking at said that the change in cost can be described by the following equation: $$∆C=∇C∙∆w$$ where $∇C$ is the gradient vector/vector derivative of the cost function (MSE) and $∆w$ is the change in weights. It said that the goal is to make the change in cost negative. Good so far. My issue is with the next part. It states that $$∆v=-η∇C$$ My issue is with this, and why $∆v$ is set to this. Why would we want to change the weights by a small amount of the gradient function?

$\endgroup$
0
$\begingroup$

Upon writing this I have realised the answer to the question. I am still going to post so that anyone else who wants to learn where the update rule comes from can do so. I have come to this by studying the equation carefully. $∇C$ is the gradient vector of the cost function. The definition of the gradient vector is a collection of partial derivatives that point in the direction of steepest ascent. Since we are performing gradient 'descent', we take the negative of this, as we hope to descend towards the minimum point. The issue for me was how this relates to the weights. It does so because we want to 'take'/'travel' along this vector towards the minimum, so we add this onto the weights. Finally, we use neta which is a small constant. It is small so that the inequality $∆C>0$ is obeyed, because we want to always decrease the cost, not increase it. However, too small, and the algorithm will take a long time to converge. This means the value for eta must be experimented with.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.