I have been trying to understand the backpropagation for a while now. I have came across two variants of it.

  1. In the Andrew Ng class the derivatives of the weights of hidden layers are calculated using the error signal that is distributed back to the hidden node.
  2. In Geoffrey Hinton class the derivatives of the weights of hidden layers are calculated using the derivatives of the next layer that are already computed and from my knowledge of calculus that makes more sense.

Can someone explain how the first variant works?


The first variant is the second variant, or more accurately there is only one type of backpropagation, and that works with the gradients of a loss function with respect to parameters of the network.

This is not an uncommon point to have questions about though, the main issue that I see causing confusion is when the loss function has been cleverly constructed so that it works with the output layer activation function, and the derivative term is numerically $\hat{y} - y$, which looks the same as taking the linear error directly. People studying the code implementing a network like this can easily come to the conclusion that the initial gradient is in fact an initial error (and whilst these are numerically equal, they are different concepts, and in a generic neural network they don't have to be equal)

This situation applies for the following network architectures:

  • Mean squared error $\frac{1}{2N}\sum_{i=1}^N(\hat{y}_i - y_i)^2$ and linear output layer - note the multiplier $\frac{1}{2}$ is there to deliberately simplify the derivative.

  • Binary cross-entropy $\frac{-1}{N}\sum_{i=1}^Ny_i\text{log}(\hat{y}_i) + (1-y_i)\text{log}(1-\hat{y}_i)$ and sigmoid output layer. The derivative of the loss neatly cancels out the derivative of the sigmoid, leaving you with gradient at the pre-transform stage of $\hat{y} - y$.

  • Multi-class logloss with one-hot encoding of true classes $\frac{-1}{N}\sum_{i=1}^N\mathbf{y}_i\cdot\text{log}(\hat{\mathbf{y}}_i)$ and softmax output layer. Again the derivative of the loss neatly cancels out, leaving you with gradient at the pre-transform stage of $\hat{y} - y$ for the true class.

So when you are told that backpropagation processes an "error signal" or "the error" backwards through the network, just mentally add "the gradient of" to the start of the phrase. Some people will say it knowingly as shorthand, others might be honestly confused.

The same applies to deeper layers, although then there is no other source for the confused "this is the error being distributed" other than as shorthand for "this is the [gradient of the] error being distributed".

  • $\begingroup$ well i get that is this true for the weights between the hidden and the output layer but is it the same case between two hidden layers as the derivative of error at that node wont be y^ - y $\endgroup$ Aug 12 '17 at 15:26
  • $\begingroup$ @lakshaytaneja: There are no more neat tricks to shortcut the derivatives deeper in the network, but backpropagation is still all about gradients, there are no "variants" where it is about error values directly. I explained the output layer here, because that in my experience is a root cause for misunderstanding. Maybe explain more where your confusion is (why you think Andrew Ng is explaining distributing error values - he is not, but perhaps the explanation is missing something that would help you) by giving a specific quote? $\endgroup$ Aug 12 '17 at 15:42
  • $\begingroup$ I think i'm clear now in his video the way he shows diagrammatically it seems like the error is being propagated back but yeah its only true for the weights between hidden and output layer after that the recursive function he gave is using derivates from the previous layer.Its just that the way he explained it for the first layer got stuck in my mind anyways thanks for clearing my doubt :) $\endgroup$ Aug 12 '17 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.