I am trying to understand of purpose of partial differentiation in NN training by knowing how to interpret gradients and their partial derivatives. Below is my way of interpreting them so I would like to know if I am correct, and if not, could someone please point me in the right direction.
If we are working with functions that depend on a single variable, then derivative of that function with respect to that particular variable is a slope (i.e. constant) which tells us how the changes in the dependent variable will effect the changes in the function value.
If we are working with functions that depend on several (N) variables, then derivative of that function with respect to all of these dependent variables is a gradient (i.e. vector of partial derivatives) which points to the direction of function extreme. Each partial derivative corresponds to a specific dimension in the N dimensional space that we are trying to optimize (e.g. quadratic cost function C(W,b)).
My question is, when we calculate partial derivative with respect to one parameter (e.g. weight between input x1 and 1st hidden layer neuron) then we are treating all other weights and biases as constants and we are evaluating how will cost function change if we were to take a step in the direction that is represented by that particular weight. Is this correct? If not, please correct my understanding of partial differentiation in NN training procedure.
Also, what is the role of Jacobian matrix in NN training?
Thank you so much!