1
$\begingroup$

Can anyone mathematically prove this equation given the values of $dz^{[2]}$, $W^{[2]}$, $z^{[1]}$ and the activation function $g^{[1]}$

$dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]})$

$\endgroup$
0
$\begingroup$

For the formal proof I think it's best to post on maths stackexchange instead.

Here are one of the best videos on backpropagation:

(link)

(link)

Your equation shows the chain rule.

  • hadamard-multiplying any matrix (like $w^T$) by a vector-with-n-entries (in your case by $dz^{[2]}$) is basically saying "I scale each stripe of my matrix by the corresponding entry in vector $dz$"

regarding the "stripes":

  • each row of $W^{[2]}$ represents a weights from one of our neurons to all neurons in preceding layer

  • each column of $W^{[2]}$ represents weights from all our neurons towards one particular neuron in preceding layer.

  • your activations ($g$) each depends on the weights that flow into it (the weights are the parameters of these activations).

  • your activations also depend on the values of neurons that flow into them. Notice, although neurons are the same entities used by the activations, they are treated as separate branches on the tree diagram (link)

If you watch the "tree diagram link", here is one more hint: in the video, the girl was summing up the terms from different branches. This is exactly what the dot product is doing during backprop.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.