1
$\begingroup$

I am deriving a Weight update for a simple toy network with a Sigmoid Output Layer. I need some help double checking my math to make sure I did it correctly.

I am using Cross-Entropy Loss as my Loss function: C = -(ylog(y_hat) + (1-y)log(1-y_hat)

Where: y_hat = sigmoid = 1 / (1 + e^-z)

Now, I have a 1 hidden layer network architecture so I am trying to update my 2nd weight matrix: enter image description here

Chain Rule derivation to Update 2nd Weight Matrix:enter image description here

Where ```Z This is the output of my hidden layer before I apply the sigmoid activation. A1 is the hidden layer activation matrix.

This is what I get so far for each part of the chain rule I derived above:

(1)

(2)

(3)

Finally with everything put together my derived partial derivative is: enter image description here

This means sigmoid - y is a scalar value (the error from my training example) and it is multiplied by the prior layer's activation matrix (A1). Does this derivation seem correct? Sorry for formatting. I don't know LaTex yet.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Yes, that's right. Just a thing to care about (assuming this is for a fully-connected NN) is that, in your case, both $A^{[1]}$ and $W^{[2]}$ are vectors. $\endgroup$
    – Javier TG
    Commented Oct 12, 2020 at 14:12
  • 1
    $\begingroup$ Great! With my single training example with say 4 hidden layer nodes my W[2] is 1x4 (row vector) and A[1] is 4x1 (column vector). I just used the matrix terminology so I could expand this idea to batch training in the future. Thank you for checking my work. $\endgroup$
    – Coldchain9
    Commented Oct 12, 2020 at 16:37
  • 2
    $\begingroup$ All right. Then if in the future $W^{[2]}$ is a matrix be careful, because then you would need to express $\partial C/\partial W^{[2]}$ as: $$ \frac{\partial C}{\partial W^{[2]}} = (\sigma - y)(A^{[1]})^T$$ $\endgroup$
    – Javier TG
    Commented Oct 12, 2020 at 17:04
  • 1
    $\begingroup$ Great! Thank you. I actually have this exact formulation derived out as well (with the transpose) so we were definitely on the same page in regards to matrix version. Thank you again for your help. $\endgroup$
    – Coldchain9
    Commented Oct 12, 2020 at 17:53

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.