1
$\begingroup$

I just read Bishop's book Pattern Recognition and Machine Learning. I read the chapter 5.3 about backpropagation, and it said that, in a general feed-forward network, each unit computes a weighted sum of its inputs of the form $$a_j=\sum\limits_{i}w_{ji}z_i$$

Then the book says that the sum in the above equation transformed by the non-linear activation function $h(.)$ to give the activation $z_j$ of unit $j$ in the form $$z_j=h(a_j)$$

I think the notation is somehow akward: suppose I want to compute $a_2$, then $$a_2=w_{21}z_1+w_{22}z_2+\dots$$

Then does $$a_2=w_{21}z_1+w_{22}h(a_2)+\dots$$ mean that the neuron $a_2$ is connected to itself?

Improve this question
$\endgroup$
1
  • $\begingroup$ No $a$ and $z$ have different subscripts $\endgroup$
    – Alex
    Commented Oct 22, 2016 at 8:17

2 Answers 2

Reset to default
3
$\begingroup$

The equations are only working for a given layer.

If you want to generalize, you need to rewrite them as, for example : $$a^l_j=\sum\limits_{i}w^l_{ji}z^{l-1}_i + b^l_j$$

Improve this answer
$\endgroup$
5
  • $\begingroup$ So what does bishop try to points out? $\endgroup$
    – Kiki Rizki Arpiandi
    Commented Oct 21, 2016 at 10:57
  • $\begingroup$ I guess he wants to simplify the equations and the notation to the maximum $\endgroup$
    – mxdbld
    Commented Oct 21, 2016 at 10:59
  • $\begingroup$ I agree with you that bishop try to simplfy the equation but with bishop equation it make z 2 term exist in the right hand side of the equation, while z_2 term in the right hand side is activated neuron from the previous layer its make, but with the sampe notation (z_2), without super script indicate that both of them has the same value, which is actualy not $\endgroup$
    – Kiki Rizki Arpiandi
    Commented Oct 21, 2016 at 11:12
  • $\begingroup$ Mathematically speaking, the notation is wrong or misleading, yes. a_2 refers to two different entities here $\endgroup$
    – mxdbld
    Commented Oct 21, 2016 at 11:28
  • $\begingroup$ Yes I agree with you the mathematical notation is misleading. $\endgroup$
    – Kiki Rizki Arpiandi
    Commented Oct 21, 2016 at 11:35
2
$\begingroup$

The output of the neuron is computed as the activation function applied to the sum directly:

$$z_3 = h(w_{21}z_1 + w_{22}z_2) $$

You can have a look at the perceptron wikipage, where there are explanations and ilustrations of this, like this one.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Yes, but in bishop book z =h(a) instead, z is activated unit $\endgroup$
    – Kiki Rizki Arpiandi
    Commented Oct 21, 2016 at 10:39
  • $\begingroup$ Yes, sorry, I used $a$ instead of $z$, now it is corrected. Note that, as Bishop points out, units are connected to one another. In this case $z_1$ and $z_2$ are inputs to $z_3$. $\endgroup$
    – noe
    Commented Oct 21, 2016 at 10:54
  • $\begingroup$ Did you mean z_2=h(w_21z_1+w_22z_2) $\endgroup$
    – Kiki Rizki Arpiandi
    Commented Oct 21, 2016 at 11:05
  • $\begingroup$ $z_2$ is input in this unit, $z_3$ is the output (the activated unit). $\endgroup$
    – noe
    Commented Oct 21, 2016 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.