I just read Bishop's book Pattern Recognition and Machine Learning. I read the chapter 5.3 about backpropagation, and it said that, in a general feed-forward network, each unit computes a weighted sum of its inputs of the form $$a_j=\sum\limits_{i}w_{ji}z_i$$

Then the book says that the sum in the above equation transformed by the non-linear activation function $h(.)$ to give the activation $z_j$ of unit $j$ in the form $$z_j=h(a_j)$$

I think the notation is somehow akward: suppose I want to compute $a_2$, then $$a_2=w_{21}z_1+w_{22}z_2+\dots$$

Then does $$a_2=w_{21}z_1+w_{22}h(a_2)+\dots$$ mean that the neuron $a_2$ is connected to itself?

  • $\begingroup$ No $a$ and $z$ have different subscripts $\endgroup$ – Alex Oct 22 '16 at 8:17

The equations are only working for a given layer.

If you want to generalize, you need to rewrite them as, for example : $$a^l_j=\sum\limits_{i}w^l_{ji}z^{l-1}_i + b^l_j$$

  • $\begingroup$ So what does bishop try to points out? $\endgroup$ – Kiki Rizki Arpiandi Oct 21 '16 at 10:57
  • $\begingroup$ I guess he wants to simplify the equations and the notation to the maximum $\endgroup$ – mxdbld Oct 21 '16 at 10:59
  • $\begingroup$ I agree with you that bishop try to simplfy the equation but with bishop equation it make z 2 term exist in the right hand side of the equation, while z_2 term in the right hand side is activated neuron from the previous layer its make, but with the sampe notation (z_2), without super script indicate that both of them has the same value, which is actualy not $\endgroup$ – Kiki Rizki Arpiandi Oct 21 '16 at 11:12
  • $\begingroup$ Mathematically speaking, the notation is wrong or misleading, yes. a_2 refers to two different entities here $\endgroup$ – mxdbld Oct 21 '16 at 11:28
  • $\begingroup$ Yes I agree with you the mathematical notation is misleading. $\endgroup$ – Kiki Rizki Arpiandi Oct 21 '16 at 11:35

The output of the neuron is computed as the activation function applied to the sum directly:

$$z_3 = h(w_{21}z_1 + w_{22}z_2) $$

You can have a look at the perceptron wikipage, where there are explanations and ilustrations of this, like this one.

  • $\begingroup$ Yes, but in bishop book z =h(a) instead, z is activated unit $\endgroup$ – Kiki Rizki Arpiandi Oct 21 '16 at 10:39
  • $\begingroup$ Yes, sorry, I used $a$ instead of $z$, now it is corrected. Note that, as Bishop points out, units are connected to one another. In this case $z_1$ and $z_2$ are inputs to $z_3$. $\endgroup$ – noe Oct 21 '16 at 10:54
  • $\begingroup$ Did you mean z_2=h(w_21z_1+w_22z_2) $\endgroup$ – Kiki Rizki Arpiandi Oct 21 '16 at 11:05
  • $\begingroup$ $z_2$ is input in this unit, $z_3$ is the output (the activated unit). $\endgroup$ – noe Oct 21 '16 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.