I am trying to see how the fundamental formulas of backpropagation from this online book work on a small example.

The author lists these 4 fundamental equations (where I believe from the reading that the cost (or loss) function C as well as the derivative of the activation function $\sigma$ are general and work with any cost or activation function):

enter image description here

If I take a small example network:

enter image description here

and assume....

  • C = $\frac{1}{2}(a^3-y)^2$ (squared error)
  • The activation (a) for each node is just the logistic function sigmoid

From here we have the following:

  • $z^2_1=w^1_{11}x_1+w^1_{12}x_2+b_1$
  • $z^2_2=w^1_{21}x_1+w^1_{22}x_2+b_2$
  • $z^2_3=w^1_{31}x_1+w^1_{32}x_2+b_3$
  • $a^2_1 =\sigma(z^2_1)$
  • $a^2_2 =\sigma(z^2_2)$
  • $a^2_3 =\sigma(z^2_3)$
  • $z^3_1=w^2_{11}a^2_1+w^2_{12}a^2_2+w^2_{13}a^2_3$
  • $a^3=\sigma(z^3_1)$

If I want to calculate $\frac{\partial C}{\partial w^2_{11}}$ it seems from the book that I should calculate $a^1_1\delta^2_1$ where $\delta^2 = ((w^{2+1})^T\delta^{2+1})\odot\sigma'(z^2) $.

Since $W^3$ does not exist I assume it is ignored and we can calculate $\delta^2_1= \delta^{2+1}_1\odot\sigma'(z^2_1)$.

$\delta^3_1 = \frac{\partial C}{\partial z^3_1} = (a^3-y)\sigma'(z^3_1)$ so $\frac{\partial C}{\partial w^2_{11}}= a^1_1*((a^3-y)\sigma'(z^3_1))*\sigma'(z^2_1) $. $a^1_1$ doesnt exist except if this would mean $x_1?$

However, if I just calculate $\frac{\partial C}{\partial w^2_{11}}$ I get $\frac{\partial }{\partial w^2_{11}}\frac{1}{2}([\sigma(w^2_{11}a^2_1+w^2_{12}a^2_2+w^2_{13}a^2_3)]-y)^2=a^2_1(a^3_1-y)\sigma'(z^3_1)$

Where am I going wrong?


1 Answer 1


My mistake was silly. I did not notice in the book that the weight matrices are numbered as the layer they are going into..so my $w^2$ above should be $w^3$ and then the result from the book is $\frac{\partial }{\partial w^3_{11}}=a^2_1(a^3_1-y)\sigma'(z^3_1)$ exactly as calculated by just doing the derivatives.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.