Do we need to add the sigmoid derivative term in the final layer's error value?

Question

I have been studying professor Andrew Ng's Machine Learning course on Coursera. Currently, I am trying to prove the formulas for backpropagation, which is mentioned in Week 5 (in this document). Clearly, it mentions that δ(L) = a(L) − y.

However, I have also came across this video on Youtube. In 4:30, it states that δ(3) = -(y - y^) * f'(z(3)), where f is the sigmoid function. Since L = 3 and y^ = a(L), this means that δ(L) = (a(L) - y) * f'(z(L)). I've also derived the equations and got exactly like in the video. But this has an additional "f'(z(L))" term, which is not the same as in Coursera. I have even implemented this equation in the programming assignment for Week 5, but it only works without the term.

So my question is why Coursera's formula didn't have the last term "f'(z(L))" and why the formula works without it? It seems quite irrational to not have the term in the formula. Can anyone help me to explain this, or correct me if I'm wrong? Any help would be greatly appreciated, thanks.

Answer 1

Both courses are correct.

For the output layer only, and depending on which activation function and which loss function are in use, sometimes the effects of $f'(z^{(L)})$ are perfectly cancelled out by the calculation of $\frac{\partial J}{\partial \hat{y}}$

This is often arranged deliberately to simplify calculations. So:

• MSE loss $(\hat{y} - y)^2$ is paired to linear activation function $f(x) = x$

• Binary cross entropy loss $-y\text{log}(\hat{y}) - (1-y)\text{log}(1-\hat{y})$ is paired to sigmoid activation function $f(x) = \frac{1}{1+e^{-x}}$

• Multi-class cross entropy loss $-\mathbf{y}\text{log}(\hat{\mathbf{y}})$ is paired to softmax activation function $f(x_i) = \frac{e^{x_i}}{\sum_j e^{x_j}}$

In all these cases, the end result is that, for the output layer logits only:

$$\frac{\partial J}{\partial z^{(L)}} = \hat{y} - y$$

Each of these results can be verified with some differentiation. E.g. it is possible to show that $\frac{d}{dx}(-y\text{log}(\hat{y}) - (1-y)\text{log}(1-\hat{y})) = \frac{1}{\hat{y}(1-\hat{y})}$

For all the other layers, you absolutely need to include the factor of $f'(z^{(l)})$ for whatever activation function is in use on the layer you just back propagated through.