Why don't we transpose $\delta^{l+1}$ in back propagation?

Question

Using this neural network as an example:

The weight matrices are then

$$ W_0=[2\times4], W_1=[4\times4], W_2=[4\times2]$$

To find the error for the last layer, we use $$ \delta^{[2]} = \nabla C \odot \sigma'(z^{[2]})$$ which makes sense. This will produce a $[1\times 2]$ vector. But to find the error in the next layer, we use $$ \delta^{[1]} = (W_2^T\delta^{[2]})\odot \sigma'(z^{[1]}) $$

This appears to try to multiply a $[4\times2]$ matrix and a $[1\times 2]$ matrix together, which is illegal. Am I just wrong about how the layers are represented? Should $z^{[n]}$ really be a $[l\times 1]$ vector? That doesn't really make sense to me, because it would be multiplied by an $[l\times m]$ matrix as the feed-forward continues. Do we just always represent $\delta^{[n]}$ as a $[l\times 1]$ vector, and the formula doesn't mention this as it's common knowledge?

What am I missing here?

(Note: these formulae are based on this book)

Answer 1

You have some wrong dimension here. Rules for Dim of weight $W^{[l]} = d^{l} * d^{l-1} $

$ W_0 = [ 4 * 2 ] $

$ W_2 = [ 2 * 4 ] $

As

$ dim (z^{[2]}) = [2 * 1] $

so is

$ \delta^{[2]}$

So

$ W_{2}^{T} \delta^{2} $ is $ [4 * 2 ] * [2 * 1] = [4 * 1 ] dimension $