Why don't we transpose $\delta^{l+1}$ in back propagation?

Using this neural network as an example: The weight matrices are then

$$W_0=[2\times4], W_1=[4\times4], W_2=[4\times2]$$

To find the error for the last layer, we use $$\delta^{} = \nabla C \odot \sigma'(z^{})$$ which makes sense. This will produce a $$[1\times 2]$$ vector. But to find the error in the next layer, we use $$\delta^{} = (W_2^T\delta^{})\odot \sigma'(z^{})$$

This appears to try to multiply a $$[4\times2]$$ matrix and a $$[1\times 2]$$ matrix together, which is illegal. Am I just wrong about how the layers are represented? Should $$z^{[n]}$$ really be a $$[l\times 1]$$ vector? That doesn't really make sense to me, because it would be multiplied by an $$[l\times m]$$ matrix as the feed-forward continues. Do we just always represent $$\delta^{[n]}$$ as a $$[l\times 1]$$ vector, and the formula doesn't mention this as it's common knowledge?

What am I missing here?

You have some wrong dimension here. Rules for Dim of weight $$W^{[l]} = d^{l} * d^{l-1}$$

$$W_0 = [ 4 * 2 ]$$

$$W_2 = [ 2 * 4 ]$$

As

$$dim (z^{}) = [2 * 1]$$

so is

$$\delta^{}$$

So

$$W_{2}^{T} \delta^{2}$$ is $$[4 * 2 ] * [2 * 1] = [4 * 1 ] dimension$$

• Really? Then how does it feed-forward the data? The input is $[1\times 2]$ or $[2 \times 1]$ if I'm wrong about layers being row-vectors. In either case, you can't multiply the input by the first weight matrix
– Zaya
Jun 23 '20 at 20:13
• $z = Wx$ $[ 4* 2] [ 2* 1] = [4 * 1]$ which is the input dim for the next layer.
– SrJ
Jun 23 '20 at 20:21
• Oh, you know what. I had the whole thing backwards. I was thinking $z=xW+b$ when it's really $z=Wx + b$. This makes sense.
– Zaya
Jun 23 '20 at 20:22
• Yes. I hope you've got it.
– SrJ
Jun 23 '20 at 20:22