1
$\begingroup$

I have been studying professor Andrew Ng's Machine Learning course on Coursera. Currently, I am trying to prove the formulas for backpropagation, which is mentioned in Week 5 (in this document). Clearly, it mentions that δ(L) = a(L) − y.

However, I have also came across this video on Youtube. In 4:30, it states that δ(3) = -(y - y^) * f'(z(3)), where f is the sigmoid function. Since L = 3 and y^ = a(L), this means that δ(L) = (a(L) - y) * f'(z(L)). I've also derived the equations and got exactly like in the video. But this has an additional "f'(z(L))" term, which is not the same as in Coursera. I have even implemented this equation in the programming assignment for Week 5, but it only works without the term.

So my question is why Coursera's formula didn't have the last term "f'(z(L))" and why the formula works without it? It seems quite irrational to not have the term in the formula. Can anyone help me to explain this, or correct me if I'm wrong? Any help would be greatly appreciated, thanks.

$\endgroup$

1 Answer 1

0
$\begingroup$

Both courses are correct.

For the output layer only, and depending on which activation function and which loss function are in use, sometimes the effects of $f'(z^{(L)})$ are perfectly cancelled out by the calculation of $\frac{\partial J}{\partial \hat{y}}$

This is often arranged deliberately to simplify calculations. So:

  • MSE loss $(\hat{y} - y)^2$ is paired to linear activation function $f(x) = x$

  • Binary cross entropy loss $-y\text{log}(\hat{y}) - (1-y)\text{log}(1-\hat{y})$ is paired to sigmoid activation function $f(x) = \frac{1}{1+e^{-x}}$

  • Multi-class cross entropy loss $-\mathbf{y}\text{log}(\hat{\mathbf{y}})$ is paired to softmax activation function $f(x_i) = \frac{e^{x_i}}{\sum_j e^{x_j}}$

In all these cases, the end result is that, for the output layer logits only:

$$\frac{\partial J}{\partial z^{(L)}} = \hat{y} - y$$

Each of these results can be verified with some differentiation. E.g. it is possible to show that $\frac{d}{dx}(-y\text{log}(\hat{y}) - (1-y)\text{log}(1-\hat{y})) = \frac{1}{\hat{y}(1-\hat{y})}$

For all the other layers, you absolutely need to include the factor of $f'(z^{(l)})$ for whatever activation function is in use on the layer you just back propagated through.

$\endgroup$
2
  • $\begingroup$ So you mean, if I used the Binary cross entropy loss cost function instead of the square error cost function, then I would get the equation exactly like in Coursera? $\endgroup$
    – Kien Pham
    Nov 14, 2018 at 15:12
  • $\begingroup$ @KienPham: Probably yes, I didn't check what the two networks are doing in detail from the video. However, you should only really do this if the network is learning a classification problem. If it is really learning a regression problem, then MSE loss may be better, even though the maths is slightly more complicated $\endgroup$ Nov 14, 2018 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.