2
$\begingroup$

Image from Hinton's course video

As seen in this image we calculate partial derivative of Error w.r.t output of the output neuron. Shouldn't it be normal derivative? Does not that particular error is determined by only that output?

$\endgroup$
  • 1
    $\begingroup$ What we need is the derivative of the error w.r.t. the weights. In other words, how the error changes when we change the weights. But due to the chain rule (illustrated in your image), we end needing the term you mention. To know how the error changes when we change weights, we first need to know how the error changes when the output changes. Then how the output changes when the weights change. It's a chain. $\endgroup$ – rcpinto Jun 2 '16 at 3:43
2
$\begingroup$

The partial derivative is used precisely because it separates concerns about how the value is calculated (from all the other parameters and outputs in the network) from how the value affects the output. This is purely by definition so you can do the calculation.

The "normal" derivative of the error function can be expressed as the sum of any "complete" set of independent partial derivatives. There are a few such sets possible - in a simple feed-forward network one for each layer plus one for all the weights combined.

So in essence you are calculating the "normal" derivative w.r.t. the network weights, but due to the nature of the problem this is done by calculating the partial derivatives in multiple steps.


Caveat: I am probably not using those maths terms 100% accurately. For instance, I am ignoring the role of the training data and treating it as a constant.

$\endgroup$
  • $\begingroup$ Let me check if I understood correctly. That E (error) term is not for one output of the network but sum of errors on all three outputs of the network. Long story short, we have only one E for the network, not E1, E2 and E3 for each output. Is that true? $\endgroup$ – ozgur Jun 2 '16 at 10:58
  • $\begingroup$ Yes there is one E to minimise, which is calculated from the outputs of the network compared to the training example labels. However, the usual way to calculate E for multiple outputs in the last layer is simply E1+E2+E3. I am not clear whether your comment refers to 1,2,3 as labels for multiple output neurons in the final layer, or for the layers? Note that if your last layer is e.g. softmax, then in general the partial derivatives are not fully independent (they are if you use the usual multiclass logloss that goes with it - which is one reason to use it) $\endgroup$ – Neil Slater Jun 2 '16 at 11:23
1
$\begingroup$

I assume that by "normal derivative" you mean $\frac{dE}{dy_j}$.

That doesn't make sense here. When you compute the derivative $\frac{dY}{dX}$ you're asking for the limiting ratio $\frac{\Delta Y}{\Delta X}$ as you make $\Delta X$ small. This only makes sense if there actually is a limit, ie. if this ratio approaches a constant as you make $\Delta X$ small.

In your case it is usually possible to find many different deltas we could apply to the weights, all of which produce the same delta on $y_j$. (In general, if there are N weights, there is an $(N-1)$-dimensional space of weight deltas, all of which have the same effect on $y_j$.) Many of these different weight deltas will have different effects on your final $E$ (via the other $y_i$'s). This means that there is no fixed limit that the ratio $\frac{\Delta E}{\Delta y_j}$ tends towards. Different weight deltas will result in wildly different values for this ratio. And so there is no "normal derivative".

You want to find how a delta for each weight individually modifies the final $E$. That tells you how much to update them. This is precisely what the partial derivative is intended to tell you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.