Please refer to an equation in Pattern recognition and Machine Learning by Bishop. My query is related to manipulation of an error term, equation 3.18: $$E_{D}(\boldsymbol{w}) = \frac{1}{2}\sum_{n=1}^N\{t_{n} - w_{0} - \sum_{j=1}^{M-1}w_{j}\phi_{j}(x_{n})\}^{2} \tag{3.18}$$ the author goes on to get: $$w_{0} = \overline{t} - \sum_{n=1}^{M-1}w_{j}\overline\phi_{j} \tag{3.19} $$ $$where \quad \overline{t} = \frac{1}{N}\sum_{n=1}^Nt_{n} \qquad \overline{\phi_j} = \frac{1}{N}\sum_{n=1}^{N}\phi_{j}(x_{n}) \tag{3.20}$$ If I take the derivate of 3.18, w.r.t $w_{0}$, set it to zero, and divide both sides by $N$, I get: $$w_{0} = \frac{1}{N}\sum_{n=1}^{N}t_{n} - \frac{1}{N}\sum_{n=1}^{N}\sum_{j=1}^{M-1}w_{j}\phi_{j}(x_{n})$$

Am I creating any error above? If so, please help. If not, my problem is related to the second term (R.H.S., above equation):

The right term, $\frac{1}{N}\sum_{n=1}^{N}\sum_{j=1}^{M-1}w_{j}\phi_{j}(x_{n})$, is the average of double sum. I can't understand how we can simply 'push' $\frac{1}{N}$ inside to get $\sum_{j=1}^{M-1}\frac{1}{N}\sum_{n=1}^{N}w_{j}\phi_{j}(x_{n})$ and eventually equation for $\overline\phi_{j}$ (ref. 3.20), since:

$$\frac{1}{2}[(1+2) + (3+5)] \quad != 1 + 3 + \frac{1}{2}(2 + 5)$$

LHS is average of sum of sums and RHS is average 'pushed' inside.


1 Answer 1


We have

$$\frac12 [(1+2)+(3+5)]=\frac12(1+3)+\frac12(2+5)$$

We can swap the order of summation of the LHS since addition is commutative and associative.

After we decided how to pair them up, we just distribute $\frac12$ into them using distributive law.

\begin{align} \frac1N \sum_{n=1}^N \sum_{j=1}^{M-1}w_j \phi_j (x_n) &=\frac1N \sum_{j=1}^{M-1}\sum_{n=1}^N w_j \phi_j (x_n) \\ &= \sum_{j=1}^{M-1}\frac1N\sum_{n=1}^N w_j \phi_j (x_n) \\ \end{align}

where the first equality is due to we are dealing with finite sum, and hence we can swap the order of additions, that is summing up the row first or column first doesn't matter.

After which, we use distributive law to bring $\frac1N$ inside the first summation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.