For backpropagation algorithm, is it true to have weight, $w$ transposed in the expression of $dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]}) $ ? Could anyone show me why $w^{[2]T}$ instead of just $w$ ?
I have read other articles, but the transposed dimension of $w$ is ignored when chain derivation is applied.
Screenshot from Andrew Ng deeplearning coursera course video:
For backpropagation algorithm, is it true to have weight, $w$ transposed in the expression of $dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]}) $ ?
Yes it is correct. You can show it is with some different arguments:
Although this is not a robust theoretical argument, checking dimensions is actually what I do in practice when I am confused when implementing NNs.
Here's the rough logic of a dimension check on the equation in your question:
$W^{[2]}$ is a $n^{[2]} \times n^{[1]}$ matrix for the forward propagation to work.
$dz^{[2]}$ is a $n^{[2]} \times 1$ column vector (because it is the gradient of cost function with respect to $z^{[2]}$, so it has to have the same dimension as it)
$dz^{[1]}$ is a $n^{[1]} \times 1$ column vector
It is $dz^{[2]}$ that is being multiplied - so the multiplying matrix must be $\text{something} \times n^{[2]}$.
The output needs to be same dimensions as $dz^{[1]}$ - so the matrix must also be $n^{[1]} \times \text{something}$.
That means the multiplying matrix must have dimensions $n^{[1]} \times n^{[2]}$ - which $W^{[2]T}$ has (and no other component of the network could match)
But why is it $W^{[2]T}$ and not some other matrix with the desired dimensions? For that we need to go back to the chain rule. Note that the expression you have written is actually two steps of the chain rule combined - first we derive $da^{[1]}$ from $dz^{[2]}$, then we derive $dz^{[1]}$ from $da^{[1]}$. It is that first step that introduces terms from $W^{[2]}$, so I will just show that. Also, deriving this directly using matrix notation is more complex, so we'll just drop into using the index values, and use the "official" partial derivative notation.
To calculate $da^{[1]}$ for a specific $i^{th}$ neuron, you have to sum over the gradients $dz^{[2]}$ for all the neurons it links to:
$$ \frac{\partial J}{\partial a_i^{[1]}} = \sum_j \frac{\partial J}{\partial z_j^{[2]}} \frac{\partial z_j^{[2]}}{\partial a_i^{[1]}} = \sum_j \frac{\partial J}{\partial z_j^{[2]}} W_{ij}^{[2]} $$
Doing this for each $i$ value in turn gets your the full $da^{[1]}$ vector.
Compare this to how the forward propagation works in the same network using the same notation and indexing:
$$z^{[2]}_j = b^{[2]}_j + \sum_i W_{ij}^{[2]} a_i^{[1]}$$
And you can see that the index used to drive the sum is different between forward and backward calculations:
When we write $W^{[2]}_{ij}$, then $i$ is index of matrix row, $j$ is index of matrix column from $W^{[2]}$.
When multiplying forward, we sum over $i$, i.e. each row of $W^{[2]}$ multiplied by each matching element of $a^{[1]}$, and this is summed up.
When you look at the gradient, you can see the equivalent sum is over $j$.
To turn that gradient calculation into a normal matrix multiply (just so the matrix notation works, and you don't have to write out the sum) you have to swap rows and columns - which is what a transpose is.
This is partly a definitions and notation issue. We could define a different kind of matrix multiply that worked with sums column-by-column. But this is not usually done, because multiplying by the transpose does the same thing.
