Confusion in backpropagation algorithm

Question

I have been trying to understand the backpropagation for a while now. I have came across two variants of it.

1. In the Andrew Ng class the derivatives of the weights of hidden layers are calculated using the error signal that is distributed back to the hidden node.
2. In Geoffrey Hinton class the derivatives of the weights of hidden layers are calculated using the derivatives of the next layer that are already computed and from my knowledge of calculus that makes more sense.

Can someone explain how the first variant works?

Answer 1

The first variant is the second variant, or more accurately there is only one type of backpropagation, and that works with the gradients of a loss function with respect to parameters of the network.

This is not an uncommon point to have questions about though, the main issue that I see causing confusion is when the loss function has been cleverly constructed so that it works with the output layer activation function, and the derivative term is numerically $\hat{y} - y$, which looks the same as taking the linear error directly. People studying the code implementing a network like this can easily come to the conclusion that the initial gradient is in fact an initial error (and whilst these are numerically equal, they are different concepts, and in a generic neural network they don't have to be equal)

This situation applies for the following network architectures:

• Mean squared error $\frac{1}{2N}\sum_{i=1}^N(\hat{y}_i - y_i)^2$ and linear output layer - note the multiplier $\frac{1}{2}$ is there to deliberately simplify the derivative.

• Binary cross-entropy $\frac{-1}{N}\sum_{i=1}^Ny_i\text{log}(\hat{y}_i) + (1-y_i)\text{log}(1-\hat{y}_i)$ and sigmoid output layer. The derivative of the loss neatly cancels out the derivative of the sigmoid, leaving you with gradient at the pre-transform stage of $\hat{y} - y$.

• Multi-class logloss with one-hot encoding of true classes $\frac{-1}{N}\sum_{i=1}^N\mathbf{y}_i\cdot\text{log}(\hat{\mathbf{y}}_i)$ and softmax output layer. Again the derivative of the loss neatly cancels out, leaving you with gradient at the pre-transform stage of $\hat{y} - y$ for the true class.

So when you are told that backpropagation processes an "error signal" or "the error" backwards through the network, just mentally add "the gradient of" to the start of the phrase. Some people will say it knowingly as shorthand, others might be honestly confused.

The same applies to deeper layers, although then there is no other source for the confused "this is the error being distributed" other than as shorthand for "this is the [gradient of the] error being distributed".