Proper derivation of dz[1] expression for backpropagation algorithm

For backpropagation algorithm, is it true to have weight, $w$ transposed in the expression of $dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]})$ ? Could anyone show me why $w^{[2]T}$ instead of just $w$ ?

I have read other articles, but the transposed dimension of $w$ is ignored when chain derivation is applied.

Screenshot from Andrew Ng deeplearning coursera course video:

For backpropagation algorithm, is it true to have weight, $w$ transposed in the expression of $dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]})$ ?

Yes it is correct. You can show it is with some different arguments:

Checking correct dimensions

Although this is not a robust theoretical argument, checking dimensions is actually what I do in practice when I am confused when implementing NNs.

Here's the rough logic of a dimension check on the equation in your question:

• $W^{[2]}$ is a $n^{[2]} \times n^{[1]}$ matrix for the forward propagation to work.

• $dz^{[2]}$ is a $n^{[2]} \times 1$ column vector (because it is the gradient of cost function with respect to $z^{[2]}$, so it has to have the same dimension as it)

• $dz^{[1]}$ is a $n^{[1]} \times 1$ column vector

• It is $dz^{[2]}$ that is being multiplied - so the multiplying matrix must be $\text{something} \times n^{[2]}$.

• The output needs to be same dimensions as $dz^{[1]}$ - so the matrix must also be $n^{[1]} \times \text{something}$.

• That means the multiplying matrix must have dimensions $n^{[1]} \times n^{[2]}$ - which $W^{[2]T}$ has (and no other component of the network could match)

Theory from item-by-item calculation, converted to matrix math

But why is it $W^{[2]T}$ and not some other matrix with the desired dimensions? For that we need to go back to the chain rule. Note that the expression you have written is actually two steps of the chain rule combined - first we derive $da^{[1]}$ from $dz^{[2]}$, then we derive $dz^{[1]}$ from $da^{[1]}$. It is that first step that introduces terms from $W^{[2]}$, so I will just show that. Also, deriving this directly using matrix notation is more complex, so we'll just drop into using the index values, and use the "official" partial derivative notation.

To calculate $da^{[1]}$ for a specific $i^{th}$ neuron, you have to sum over the gradients $dz^{[2]}$ for all the neurons it links to:

$$\frac{\partial J}{\partial a_i^{[1]}} = \sum_j \frac{\partial J}{\partial z_j^{[2]}} \frac{\partial z_j^{[2]}}{\partial a_i^{[1]}} = \sum_j \frac{\partial J}{\partial z_j^{[2]}} W_{ij}^{[2]}$$

Doing this for each $i$ value in turn gets your the full $da^{[1]}$ vector.

Compare this to how the forward propagation works in the same network using the same notation and indexing:

$$z^{[2]}_j = b^{[2]}_j + \sum_i W_{ij}^{[2]} a_i^{[1]}$$

And you can see that the index used to drive the sum is different between forward and backward calculations:

• When we write $W^{[2]}_{ij}$, then $i$ is index of matrix row, $j$ is index of matrix column from $W^{[2]}$.

• When multiplying forward, we sum over $i$, i.e. each row of $W^{[2]}$ multiplied by each matching element of $a^{[1]}$, and this is summed up.

• When you look at the gradient, you can see the equivalent sum is over $j$.

• To turn that gradient calculation into a normal matrix multiply (just so the matrix notation works, and you don't have to write out the sum) you have to swap rows and columns - which is what a transpose is.

This is partly a definitions and notation issue. We could define a different kind of matrix multiply that worked with sums column-by-column. But this is not usually done, because multiplying by the transpose does the same thing.

• "Summing over columns at a time instead of rows when performing the multiplications is the same operation as transposing the matrix and summing over rows (i.e. a standard matrix multiply)." >> I do not understand this sentence. – kevin Oct 5 '17 at 10:34
• @kevin: I'll try to re-phrase it in the answer. Basically $i$ is index of matrix row, $j$ is index of matrix column from $W$. When multiplying forward, we sum over rows of $W$ multiplied by each matching element of $a$. When you look at the gradient, you can see the sum is over $j$. To turn that gradient calculation into a normal matrix multiply you have to swap rows and columns - which is what a transpose it. – Neil Slater Oct 5 '17 at 10:42
• Sorry for repeating my comments over and over again. I am a bit confused on "normal matrix multiply" for the gradient calculation – kevin Oct 5 '17 at 16:05
• @kevin: It is similar, but it is a different step in backprop and in this case you are building a matrix of gradients, not a vector. You can show it is correct dimensionally and using the item-by-item theory in a very similar way to this answer. But that would double the size of the answer here. I suggest you give it a try yourself to see if you have truly absorbed the answer . . . – Neil Slater Oct 6 '17 at 9:39
• Your expression is not correct. Each item in the weights matrix is indexed by $i$ and $j$ so you want $\frac{\partial J}{\partial W^{[2]}_{ij}}$. Also, there is no sum over multiple connections here, each $W_{ij}$ only connects one $a$ to one $z$, and influences the gradient of the loss function once. $a_{ij}$ does not exist, $a$ is a vector and has only one indexed dimension. The correct equation is in the course materials – Neil Slater Oct 6 '17 at 9:54