# What does this expression from gradient descent mean?

I am looking over some neural network theory and came across this equation, coupled with this description (gradient descent ball-valley analogy): ''let's think about what happens when we move the ball a small amount Δv1 in the v1 direction, and a small amount Δv2 in the v2 direction. Calculus tells us that C changes as follows:'' I don't understand where this expression has come from. It looks like the product rule? Could someone either explain what it means and/or state the area of calculus this belongs to, so I can read up on it? Many thanks!

• Do you know about multivariable calculus and partial derivatives? – timleathart Jul 4 '19 at 0:55
• @timleathart Not much... what parts would I need to know about to better understand neural networks? – Finn Williams Jul 4 '19 at 0:56
• When we are trying to work out the derivative of a multivariable function (that is, a function with more than one input variable like $C(v_1, v_2)$ above) we need to work out the derivatives for each input variable and combine them somehow. I would watch a few videos or read some introductory notes about partial derivatives if I were you. Khan Academy has some good free resources online. – timleathart Jul 4 '19 at 1:22

After reading up on multivariate calculus, I have understood what the above equation means. It is a simplified version of the multivariate chain rule - a rule used to differentiate multivariate functions. Here is what it looks like: $$\cfrac{dC}{dt}≈\cfrac{∂C}{∂v_1} \cfrac{dv_1}{dt}+\cfrac{∂C}{∂v_2} \cfrac{dv_2}{dt}$$
Time applies here because we are studying how the cost changes over time/over the course of training. Multiplying each side by $$dt$$ gives: $$dC≈\cfrac{∂C}{∂v_1} dv_1+\cfrac{∂C}{∂v_2} dv_2$$
And since the differential operator $$d$$ means a 'small change', we can replace this with delta to represent the same thing, thus arriving at the equation from the question: $$\Delta C≈\cfrac{∂C}{∂v_1} \Delta v_1+\cfrac{∂C}{∂v_2} \Delta v_2$$ To summarise, this equation is a simplified version of the multivariate chain rule, which although means the same as the original, does a better job in communicating that a 'small change' in $$v_1$$ and $$v_2$$ results in a 'small change' in $$C$$.