# Does the Bishop book imply that a neuron feeds to itself in chapter 5.3?

I just read Bishop's book Pattern Recognition and Machine Learning. I read the chapter 5.3 about backpropagation, and it said that, in a general feed-forward network, each unit computes a weighted sum of its inputs of the form $$a_j=\sum\limits_{i}w_{ji}z_i$$

Then the book says that the sum in the above equation transformed by the non-linear activation function $h(.)$ to give the activation $z_j$ of unit $j$ in the form $$z_j=h(a_j)$$

I think the notation is somehow akward: suppose I want to compute $a_2$, then $$a_2=w_{21}z_1+w_{22}z_2+\dots$$

Then does $$a_2=w_{21}z_1+w_{22}h(a_2)+\dots$$ mean that the neuron $a_2$ is connected to itself?

• No $a$ and $z$ have different subscripts
– Alex
Oct 22, 2016 at 8:17

The equations are only working for a given layer.

If you want to generalize, you need to rewrite them as, for example : $$a^l_j=\sum\limits_{i}w^l_{ji}z^{l-1}_i + b^l_j$$

• So what does bishop try to points out? Oct 21, 2016 at 10:57
• I guess he wants to simplify the equations and the notation to the maximum Oct 21, 2016 at 10:59
• I agree with you that bishop try to simplfy the equation but with bishop equation it make z 2 term exist in the right hand side of the equation, while z_2 term in the right hand side is activated neuron from the previous layer its make, but with the sampe notation (z_2), without super script indicate that both of them has the same value, which is actualy not Oct 21, 2016 at 11:12
• Mathematically speaking, the notation is wrong or misleading, yes. a_2 refers to two different entities here Oct 21, 2016 at 11:28
• Yes I agree with you the mathematical notation is misleading. Oct 21, 2016 at 11:35

The output of the neuron is computed as the activation function applied to the sum directly:

$$z_3 = h(w_{21}z_1 + w_{22}z_2)$$

You can have a look at the perceptron wikipage, where there are explanations and ilustrations of this, like this one.

• Yes, but in bishop book z =h(a) instead, z is activated unit Oct 21, 2016 at 10:39
• Yes, sorry, I used $a$ instead of $z$, now it is corrected. Note that, as Bishop points out, units are connected to one another. In this case $z_1$ and $z_2$ are inputs to $z_3$.
– noe
Oct 21, 2016 at 10:54
• Did you mean z_2=h(w_21z_1+w_22z_2) Oct 21, 2016 at 11:05
• $z_2$ is input in this unit, $z_3$ is the output (the activated unit).
– noe
Oct 21, 2016 at 12:06