Justification for values used in backpropagation

Question

I'm learning the method for backpropagation in adjusting weights. A generalization of a formula used to determine the change made to a respective weight is

where is the rate the total error changes as the i-th weight changes. I get why this value is interesting in the context of what we're trying to accomplish with backpropagation, but I don't totally understand why this is a good value to use when determining how to adjust the new weight, particularly when there's multiple weights influencing the total error. Why is this a "good" number to use to when changing our weight?

Answer 1

The objective of back-propagation is to isolate the effect of each weight on the total error. Once the effect is isolated, each weight can be changed individually such the total error is minimized.

During back-propagation, the effects of other weights are automatically isolated. Take, for example, the function:

$$ E_{total}(x) = w_1x + w_2x^2 \\ $$

Then

$$ dE_{total}(x)/dw_1 = d(w_1x)/dw_1 + d(w_2x^2)/dw_1 \\ dE_{total}(x)/dw_1 =x + 0 $$

So, you see, back-propagation ignores the effects of other weights by definition of differentiation. When you back-propagate with respect to $w_1$, all other $w$s are considered constant.

$dE_{total}/dw_i$ is the change in error as you change a particular weight $w_i$ and only that weight. So it makes sense that you would want to change $w_i$ in the opposite direction of that change i.e. $-dE_{total}/dw_i$.

Answer 2

In order to understand why it is a good value to use when determining new weight, we have to understand the maths behind backpropagation and what happens by updating weights at each iteration.

Backpropagation uses the chain rule method to calculate the new weights. At each iteration, the weights are updated with the hope that we are converging towards an optimal set of weights were the neural network performs best. In other words, by updating the weight at each iteration backpropagation is searching for optimal parameters in a very high dimensional space.

Each parameter's influence on total error can differ at every iteration. But by repeating this process over a set of training data after multiple epochs, backpropagation is able to optimize all those parameters (to some extent). The parameters which have less influence on error will change less often by definition of the chain rule.