Specifically, I am wondering about summing the partial derivative of error in relation to the previous node's output for all superseding layers.

My network is a 4 layer feed forward network; layer one has a theta matrix of 23950x39916, layer two 39916x50560, layer 3 50560x39916, and layer four 39916x23950. I am using swift (x*sigmoid(x)) for the activation in every layer. I derived the following equations for the delta of the weights for each layer: https://i.imgur.com/V4eQq6D.png

For context, delta w = δ*a(L-1) so (delta w*w)/a(L-1)= δ*w, which is what I have seen in every place I've looked in terms of calculating the delta of a layer before the output. Again, my question is about why that pdf I linked says that instead of just δ*w for the superseding layer I need to include the sum of δ*w for every superseding layer.

I don't know how to do equation formatting, so everything is at that link. Here is the document I reference: https://www.cs.swarthmore.edu/~meeden/cs81/s10/BackPropDeriv.pdf

edit: I corrected the weights (you'll notice they don't have the correct subscripts) but I still don't know about summing them.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.