I am looking over some neural network theory and came across this equation, coupled with this description (gradient descent ball-valley analogy):

''let's think about what happens when we move the ball a small amount Δv1 in the v1 direction, and a small amount Δv2 in the v2 direction. Calculus tells us that C changes as follows:''

I don't understand where this expression has come from. It looks like the product rule? Could someone either explain what it means and/or state the area of calculus this belongs to, so I can read up on it? Many thanks!

  • 2
    $\begingroup$ Do you know about multivariable calculus and partial derivatives? $\endgroup$ – timleathart Jul 4 '19 at 0:55
  • $\begingroup$ @timleathart Not much... what parts would I need to know about to better understand neural networks? $\endgroup$ – Finn Williams Jul 4 '19 at 0:56
  • 2
    $\begingroup$ When we are trying to work out the derivative of a multivariable function (that is, a function with more than one input variable like $C(v_1, v_2)$ above) we need to work out the derivatives for each input variable and combine them somehow. I would watch a few videos or read some introductory notes about partial derivatives if I were you. Khan Academy has some good free resources online. $\endgroup$ – timleathart Jul 4 '19 at 1:22

After reading up on multivariate calculus, I have understood what the above equation means. It is a simplified version of the multivariate chain rule - a rule used to differentiate multivariate functions. Here is what it looks like: $$\cfrac{dC}{dt}≈\cfrac{∂C}{∂v_1} \cfrac{dv_1}{dt}+\cfrac{∂C}{∂v_2} \cfrac{dv_2}{dt}$$

Time applies here because we are studying how the cost changes over time/over the course of training. Multiplying each side by $dt$ gives: $$dC≈\cfrac{∂C}{∂v_1} dv_1+\cfrac{∂C}{∂v_2} dv_2$$

And since the differential operator $d$ means a 'small change', we can replace this with delta to represent the same thing, thus arriving at the equation from the question: $$\Delta C≈\cfrac{∂C}{∂v_1} \Delta v_1+\cfrac{∂C}{∂v_2} \Delta v_2$$ To summarise, this equation is a simplified version of the multivariate chain rule, which although means the same as the original, does a better job in communicating that a 'small change' in $v_1$ and $v_2$ results in a 'small change' in $C$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.