1
$\begingroup$

Using this neural network as an example:

NN Example

The weight matrices are then

$$ W_0=[2\times4], W_1=[4\times4], W_2=[4\times2]$$

To find the error for the last layer, we use $$ \delta^{[2]} = \nabla C \odot \sigma'(z^{[2]})$$ which makes sense. This will produce a $[1\times 2]$ vector. But to find the error in the next layer, we use $$ \delta^{[1]} = (W_2^T\delta^{[2]})\odot \sigma'(z^{[1]}) $$

This appears to try to multiply a $[4\times2]$ matrix and a $[1\times 2]$ matrix together, which is illegal. Am I just wrong about how the layers are represented? Should $z^{[n]}$ really be a $[l\times 1]$ vector? That doesn't really make sense to me, because it would be multiplied by an $[l\times m]$ matrix as the feed-forward continues. Do we just always represent $\delta^{[n]}$ as a $[l\times 1]$ vector, and the formula doesn't mention this as it's common knowledge?

What am I missing here?

(Note: these formulae are based on this book)

$\endgroup$
1
$\begingroup$

You have some wrong dimension here. Rules for Dim of weight $W^{[l]} = d^{l} * d^{l-1} $

$ W_0 = [ 4 * 2 ] $

$ W_2 = [ 2 * 4 ] $

As

$ dim (z^{[2]}) = [2 * 1] $

so is

$ \delta^{[2]}$

So

$ W_{2}^{T} \delta^{2} $ is $ [4 * 2 ] * [2 * 1] = [4 * 1 ] dimension $

$\endgroup$
4
  • $\begingroup$ Really? Then how does it feed-forward the data? The input is $[1\times 2]$ or $[2 \times 1]$ if I'm wrong about layers being row-vectors. In either case, you can't multiply the input by the first weight matrix $\endgroup$
    – Zaya
    Jun 23 '20 at 20:13
  • 1
    $\begingroup$ $ z = Wx $ $ [ 4* 2] [ 2* 1] = [4 * 1] $ which is the input dim for the next layer. $\endgroup$
    – SrJ
    Jun 23 '20 at 20:21
  • $\begingroup$ Oh, you know what. I had the whole thing backwards. I was thinking $z=xW+b$ when it's really $z=Wx + b$. This makes sense. $\endgroup$
    – Zaya
    Jun 23 '20 at 20:22
  • $\begingroup$ Yes. I hope you've got it. $\endgroup$
    – SrJ
    Jun 23 '20 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.