How to calculate new weights for neurons - what is the general equation for it?

closed as too broad by Stephen Rauch, Mephy, oW_, BrunoGL, tuomastik Aug 11 at 5:29

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • There isn't one general equation. If you have a fixed generic architecture though (e.g. a feed-forward network with generic layer sizes, generic cost function and generic transfer function on each layer), it is possible to specify generic equations for back propagation in order to calculate the gradient. Once you have the gradient, you need to specify your optimiser before you can write the equations for updating the weights. If you can add some context to your question, and maybe explain why the many easy-to-find examples are not correct for you, it would make the question better, – Neil Slater Aug 10 at 11:45
  • @NeilSlater OK, but I mean "general" for for example "derivative of act_func * learn_rate etc. *etc." – Pan Mikuś Aug 10 at 18:52
  • Yes that's what I mean too. Please edit your question to fix the architecture and optimisation method - the simplest is a feed-forward network consisting of fully connected layers, and basic gradient descent. Practically, it is not possible to obtain a single general equation that covers specialist layers such as dropout, pooling in CNNs or LSTM cells (I guess it is possible to put all that into math notation and call it an equation, but it would be very unwieldy) – Neil Slater Aug 10 at 19:08

I can point you to a good resource http://blog.kaggle.com/2017/12/06/introduction-to-neural-networks-2/

I'm assuming you already understand forward propagation.(Initialize the weights randomly, calculate the net input and use an activation function over the net input to get the output and then forward the output to another layer and repeat the process).

The intuition behind back propagation is to gradually update weights that is optimizing your loss function. When you have an output, you run it through a Loss function L and find the loss. Then let's say your optimization function is Gradient Descent, you determine how much the current loss will change with respect to a small change in each of the weights. You calculate the gradient of L(partially differentiate L) with respect to every weight in the network. Then you take a small step(learning rate) in the negative direction of you gradient. And Repeat this process until you've optimized your loss function.

So the equation you're asking for really depends on which optimizing algorithm you're using to optimize your weights. Follow the link and it'll walk you through each step of the back propagation using Gradient Descent.

Not the answer you're looking for? Browse other questions tagged or ask your own question.