I am trying to see how the fundamental formulas of backpropagation from this online book work on a small example.
The author lists these 4 fundamental equations (where I believe from the reading that the cost (or loss) function C as well as the derivative of the activation function $\sigma$ are general and work with any cost or activation function):
If I take a small example network:
and assume....
- C = $\frac{1}{2}(a^3-y)^2$ (squared error)
- The activation (a) for each node is just the logistic function sigmoid
From here we have the following:
- $z^2_1=w^1_{11}x_1+w^1_{12}x_2+b_1$
- $z^2_2=w^1_{21}x_1+w^1_{22}x_2+b_2$
- $z^2_3=w^1_{31}x_1+w^1_{32}x_2+b_3$
- $a^2_1 =\sigma(z^2_1)$
- $a^2_2 =\sigma(z^2_2)$
- $a^2_3 =\sigma(z^2_3)$
- $z^3_1=w^2_{11}a^2_1+w^2_{12}a^2_2+w^2_{13}a^2_3$
- $a^3=\sigma(z^3_1)$
If I want to calculate $\frac{\partial C}{\partial w^2_{11}}$ it seems from the book that I should calculate $a^1_1\delta^2_1$ where $\delta^2 = ((w^{2+1})^T\delta^{2+1})\odot\sigma'(z^2) $.
Since $W^3$ does not exist I assume it is ignored and we can calculate $\delta^2_1= \delta^{2+1}_1\odot\sigma'(z^2_1)$.
$\delta^3_1 = \frac{\partial C}{\partial z^3_1} = (a^3-y)\sigma'(z^3_1)$ so $\frac{\partial C}{\partial w^2_{11}}= a^1_1*((a^3-y)\sigma'(z^3_1))*\sigma'(z^2_1) $. $a^1_1$ doesnt exist except if this would mean $x_1?$
However, if I just calculate $\frac{\partial C}{\partial w^2_{11}}$ I get $\frac{\partial }{\partial w^2_{11}}\frac{1}{2}([\sigma(w^2_{11}a^2_1+w^2_{12}a^2_2+w^2_{13}a^2_3)]-y)^2=a^2_1(a^3_1-y)\sigma'(z^3_1)$
Where am I going wrong?
