1
$\begingroup$

I was going through the derivation of backpropagation algorithm provided in this document (adding just for reference). I have doubt at one specific point in this derivation. The derivation goes as follows:

Notation:

  1. The subscript $k$ denotes the output layer
  2. The subscript $j$ denotes the hidden layer
  3. The subscript $i$ denotes the input layer
  4. $w_{kj}$ denotes a weight from the hidden to the output layer
  5. $w_{ji}$ denotes a weight from the input to the hidden layer
  6. $a$ denotes an activation value
  7. $t$ denotes a target value
  8. $net$ denotes the net input

The total error in a network is given by the following equation $$E=\frac12 \sum_{k}(t_k-a_k)^2$$ We want to adjust the network’s weights to reduce this overall error: $$\Delta W \propto -\frac{\partial E}{\partial W}$$ We will begin at the output layer with a particular weight. $$\Delta w_{kj} \propto -\frac{\partial E}{\partial w_{kj}}$$ However error is not directly a function of a weight. We expand this as follows. $$\Delta w_{kj} = -\epsilon \frac{\partial E}{\partial a_k} \frac{\partial a_k}{\partial net_k} \frac{\partial net_k}{\partial w_{kj}}$$ Now the document individually calculates above three partial derivatives to get following final equation for weight change rule for a hidden to output weight [Section 4.4 in the attached document]: $$\Delta w_{kj} = \epsilon (t_k-a_k)a_k(1-a_k)a_j$$ or, $$\Delta w_{kj} = \epsilon \delta_k a_j$$ I followed till above equation.

Now, the document continues the derivation as follows (I am adding exact wordings from the document) [Section 4.5 in the document attached]

Weight change rule for an input to hidden weight

Now we have to determine the appropriate weight change for an input to hidden weight. This is more complicated because it depends on the error at all of the nodes this weighted connection can lead to. $$\Delta w_{ji} \propto -[\sum_{k}\frac{\partial E}{\partial a_k} \frac{\partial a_k}{\partial net_k} \frac{\partial net_k}{\partial a_j}]\frac{\partial a_j}{\partial net_j}\frac{\partial net_j}{\partial w_{ji}}$$

I couldn't follow that why there is this extra $\sum_{k}$ in above equation as we have already considered sum over all $k$ when we wrote error $E=\frac12 \sum_{k}(t_k-a_k)^2$ Any lights?

$\endgroup$
0
$\begingroup$

For a while I thought you were right, but the equation is correct.

If you think about it conceptually, then I think you will see that it has to be correct. When updating $w_{ij}$, you have to make the change that improves the total error $E$, and not just the error $E_k$ of output node $k$, right? So you have to be looking for the weights that give you the lowest $E$, that is, where $\partial E / \partial a_k$ is 0.

If you write out the sum over $k$ in the equation for $E$, you get one term for $a_1$, one for $a_2$, and so on. The sum over $k$ in the equation that confuses you, just makes sure to first get the $a_1$ term from $E$, then the $a_2$ term, and so in. It first differentiates $E$ to $a_1$, which is $t_1 + a_1$ if I did that correctly, and gets the correct tion for that. Then for $a_2$, until the end, and finally you have to sum over $k$ to add up the corrections from all terms.

$\endgroup$
  • $\begingroup$ Sorry, I still couldn't follow. I kind of gets feel that this is right but am unable to generate strong intuition for it as I get for the first $\sum_k$. If I write $E$ in expanded form, it is $$E=\frac12 [(t_1-a_1)^2 + (t_2-a_2)^2 +...+ (t_k-a_k)^2]$$. Then $$\frac{\partial E}{\partial a_k} = (t_k-a_k)$$. Now why do we again need to do sum over $k$ as the current $E$ above already encompasses error for each node in the output layer? $\endgroup$ – ashutosh Nov 10 at 17:41
  • $\begingroup$ Because you need to optimize for the total error, E. E_k is the error measured on a single output node. $\endgroup$ – Paul Nov 23 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.