3
$\begingroup$

I am trying to understand of purpose of partial differentiation in NN training by knowing how to interpret gradients and their partial derivatives. Below is my way of interpreting them so I would like to know if I am correct, and if not, could someone please point me in the right direction.

If we are working with functions that depend on a single variable, then derivative of that function with respect to that particular variable is a slope (i.e. constant) which tells us how the changes in the dependent variable will effect the changes in the function value.

If we are working with functions that depend on several (N) variables, then derivative of that function with respect to all of these dependent variables is a gradient (i.e. vector of partial derivatives) which points to the direction of function extreme. Each partial derivative corresponds to a specific dimension in the N dimensional space that we are trying to optimize (e.g. quadratic cost function C(W,b)).

My question is, when we calculate partial derivative with respect to one parameter (e.g. weight between input x1 and 1st hidden layer neuron) then we are treating all other weights and biases as constants and we are evaluating how will cost function change if we were to take a step in the direction that is represented by that particular weight. Is this correct? If not, please correct my understanding of partial differentiation in NN training procedure.

Also, what is the role of Jacobian matrix in NN training?

Thank you so much!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

My question is, when we calculate partial derivative with respect to one parameter (e.g. weight between input x1 and 1st hidden layer neuron) then we are treating all other weights and biases as constants and we are evaluating how will cost function change if we were to take a step in the direction that is represented by that particular weight. Is this correct?

Yes.

Long answer: that is exactly the meaning of partial derivative. It's the impact of a specific variable, while keeping all the others constant. It's an all things equal condition (or ceteris paribus, if you like Latin).

You need it in order to understand how much that specific parameter, at that current value, is contributing to the final Loss. You have to keep everything else equal in order to understand what is that parameter's responsibility of the final model's error.

Also, what is the role of Jacobian matrix in NN training?

More generally, the Jacobian matrix of a Neural Network is a matrix of the partial derivatives of the y. It's size is:

( Number of observations, Number of parameters )

It's a way to store/represent your gradient information. The application of the chain rule to execute backprop is executed on those in practice.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you. I really like your answer. Could you possibly provide little bit more details on Jacobian Matrix, or point me to some helpful resource. Thanks! $\endgroup$
    – Stefan Radonjic
    Jan 17, 2020 at 19:47
  • 1
    $\begingroup$ The Jacobian matrix is that object where information is stored during training, in order to compute gradients at each iteration. This article shows how to calculate it using several Python libraries. This paper is more technical, but goes deeper. $\endgroup$
    – Leevo
    Jan 18, 2020 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.