1
$\begingroup$

Using this neural network as an example:

NN Example

The weight matrices are then

$$ W_0=[2\times4], W_1=[4\times4], W_2=[4\times2]$$

To find the error for the last layer, we use $$ \delta^{[2]} = \nabla C \odot \sigma'(z^{[2]})$$ which makes sense. This will produce a $[1\times 2]$ vector. But to find the error in the next layer, we use $$ \delta^{[1]} = (W_2^T\delta^{[2]})\odot \sigma'(z^{[1]}) $$

This appears to try to multiply a $[4\times2]$ matrix and a $[1\times 2]$ matrix together, which is illegal. Am I just wrong about how the layers are represented? Should $z^{[n]}$ really be a $[l\times 1]$ vector? That doesn't really make sense to me, because it would be multiplied by an $[l\times m]$ matrix as the feed-forward continues. Do we just always represent $\delta^{[n]}$ as a $[l\times 1]$ vector, and the formula doesn't mention this as it's common knowledge?

What am I missing here?

(Note: these formulae are based on this book)

$\endgroup$

1 Answer 1

1
$\begingroup$

You have some wrong dimension here. Rules for Dim of weight $W^{[l]} = d^{l} * d^{l-1} $

$ W_0 = [ 4 * 2 ] $

$ W_2 = [ 2 * 4 ] $

As

$ dim (z^{[2]}) = [2 * 1] $

so is

$ \delta^{[2]}$

So

$ W_{2}^{T} \delta^{2} $ is $ [4 * 2 ] * [2 * 1] = [4 * 1 ] dimension $

$\endgroup$
4
  • $\begingroup$ Really? Then how does it feed-forward the data? The input is $[1\times 2]$ or $[2 \times 1]$ if I'm wrong about layers being row-vectors. In either case, you can't multiply the input by the first weight matrix $\endgroup$
    – Zaya
    Commented Jun 23, 2020 at 20:13
  • 1
    $\begingroup$ $ z = Wx $ $ [ 4* 2] [ 2* 1] = [4 * 1] $ which is the input dim for the next layer. $\endgroup$
    – SrJ
    Commented Jun 23, 2020 at 20:21
  • $\begingroup$ Oh, you know what. I had the whole thing backwards. I was thinking $z=xW+b$ when it's really $z=Wx + b$. This makes sense. $\endgroup$
    – Zaya
    Commented Jun 23, 2020 at 20:22
  • $\begingroup$ Yes. I hope you've got it. $\endgroup$
    – SrJ
    Commented Jun 23, 2020 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.