2
$\begingroup$

I use chain rule when doing backpropagation and then I do Gradient Descent with weighting coefficient and I am updating the weight, so I do not understand how the method works in the equations below.

\begin{align*} Z^{[1]}&=W^{[1]}X+b^{[1]} \\ A^{[1]}&=g^{[1]}\left(Z^{[1]}\right) \\ Z^{[2]}&=W^{[2]}A^{[1]}+b^{[2]} \\ A^{[2]}&=g^{[2]}\left(Z^{[2]}\right) \\ \vdots & \\ A^{[L]}&=g^{[L]}\left(Z^{[L]}\right)=\hat{Y} \\ \\ dZ^{[L]}&=A^{[L]}-Y \\ dW^{[L]}&=\frac1m\,dZ^{[L]}{A^{[L]}}^{T} \\ db^{[L]}&=\frac1m \,\;\operatorname{np.sum}\left(dZ^{[L]},\;\operatorname{axis}=1,\;\operatorname{keepdims}=\operatorname{True} \right) \\ dZ^{[L-1]}&={dW^{[L]}}^{T}dZ^{[2]}g'^{[1]}\left(Z^{[1]}\right) \\ dW^{[1]}&=\frac1m\,dZ^{[1]}{A^{[1]}}^{T} \\ db^{[1]}&=\frac1m \,\;\operatorname{np.sum}\left(dZ^{[1]},\;\operatorname{axis}=1,\;\operatorname{keepdims}=\operatorname{True} \right). \end{align*}

(Source Image)

$\endgroup$
  • 4
    $\begingroup$ Hi. Unfortunately all I get from your question is that you do not understand the pseudocode in the linked image. Please can you give more details, and maybe more context about what you do not understand, so someone can help you. At the moment it is not clear, and also one very good (probably the best) explanation is freely available in the course that the image is from: coursera.org/learn/neural-networks-deep-learning $\endgroup$ – Neil Slater Sep 11 '18 at 12:42
  • $\begingroup$ @NeilSlater .Thank you, I understand.I want to ask you again to be sure: then these formulas are derived from chain rules. $\endgroup$ – Engin Sep 11 '18 at 16:20
  • $\begingroup$ Yes the second set of formulae/pseudo-code starting from $dZ^{[L]}$ are differentiation and the chain rule applied to the first set of formulae. Although the cost function is not shown, so you don't have the derivation of initial $dZ^{[L]}$ (the value is what you expect from a bunch of standard NN setups though) $\endgroup$ – Neil Slater Sep 11 '18 at 16:24
  • $\begingroup$ You need to understand Matrix/Vector Calculus a little bit and then everything will be easier and Intuitive as well $\endgroup$ – Aditya Sep 14 '18 at 4:22
1
$\begingroup$

The equations of back propagation that you have mentioned can be derived from the chain rule. I see from your comment that you already understand them and just want a double check on this statement. So I shall give a very quick derivation of the equations.

Look at the following computation graph: Computation graph for logistic regression

I shall consider one training example for now. You can extend the concept to the entire training set in a similar way.

The loss function depicted here is: $$-ylog(a) - (1 - y)log(1 - a)$$. This simply means you are highly penalising the prediction of a = 1 when y = 0.

Differentiating this function with respect to a:

$\frac{dL}{da}$ = $-\frac{y}{a} - (1 - y).\frac{1}{1 - a}.(-1)$ OR $\frac{dL}{da}$ = $-\frac{y}{a} +\frac{1 - y}{1 - a}$

This is because $\frac{d f(g(x))}{dx} = f'(x). g'(x)$. Here $g(x) = 1 - a, f(x) = log (x).$ Therefore, $f(g(x)) = log(g(x)) = log( 1 - a)$.

Now, according to the computation graph, $a = sigmoid(z)$ or $a = \frac{1}{1 + e^{-z}}$. Utilising the same rules of calculus as done above,

$\frac{da}{dz}$ = $a(1 - a)$

Now to calculate $\frac{dL}{dz}$

$\frac{dL}{dz}$ = $\frac{dL}{da}.\frac{da}{dz}$

$\frac{dL}{dz}$ = [$-\frac{y}{a} +\frac{1 - y}{1 - a}$] . [$a(1 - a)$]

When you go ahead and solve this, you get the result as $a - y$ (exactly the first equation of back propagation).

Similarly, the second equation (or $\frac{dL}{dw}$) can be calculated as $\frac{dL}{dz}.\frac{dz}{dw}$. You already have the first term as $a - y$. To calculate the second term,

$\frac{dz}{dw} = x$. This can be simply understood primarily because z is a linear equation on being parametrised by w. And the bias unit is a constant (hence it equals 0 on differentiation).

And that is how you can go on back propagating.

A bit of implementation note though. When you consider all training examples at once, you need to introduce certain subtleties like np.sum(), or the pretext of $\frac{1}{m}$ or the various transpose operations on activations $A^{[l]}$. I guess you should now be able to handle that, given the derivations above.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.