0
$\begingroup$

Specifically, I am wondering about summing the partial derivative of error in relation to the previous node's output for all superseding layers.

My network is a 4 layer feed forward network; layer one has a theta matrix of 23950x39916, layer two 39916x50560, layer 3 50560x39916, and layer four 39916x23950. I am using swift (x*sigmoid(x)) for the activation in every layer. I derived the following equations for the delta of the weights for each layer: https://i.imgur.com/V4eQq6D.png

For context, delta w = δ*a(L-1) so (delta w*w)/a(L-1)= δ*w, which is what I have seen in every place I've looked in terms of calculating the delta of a layer before the output. Again, my question is about why that pdf I linked says that instead of just δ*w for the superseding layer I need to include the sum of δ*w for every superseding layer.

I don't know how to do equation formatting, so everything is at that link. Here is the document I reference: https://www.cs.swarthmore.edu/~meeden/cs81/s10/BackPropDeriv.pdf

edit: I corrected the weights (you'll notice they don't have the correct subscripts) but I still don't know about summing them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.