To understand how neural networks, and backpropagation are actually working, I've built a small program to do the calculations, but something is definitely wrong, as my weights are the same after gradient descent. In this example the inputs will have two neurons, the outputs will also have two neurons, there will be two hidden layers, with 3 neurons each. I'll work with two observations. I'll initialise both weights, and biases with 1.
(I'm new at machine learning, and super confused with the different notations, and formatting, indexes, so to clarify, in case of activations, the upper index will show the layers index which is in, the lower index before will show the index of the observation, and the lower index after will show the index of it in the layer, so the third neuron in the second layer in the first observation will look like $_0a^1_2$ (indexes are 0 starting). In weights, the upper index will show which is the index of layer it's "going to", the first index of the lower index will show which is the index of neuron it's "coming from", and the second index of the lower index will show which is the index of neuron it's "going to", so the third layers first neurons weight, which connects it with the second layers first neuron will look like $w^1_{0,0}$.)
As far as I learned: $a_n = \sigma (a^{n-1} * w^{n-1} + b^{n-1})$ where $n$ is the index of the layer, and where $*$ is actually matrix multiplication.
In other visualisation, calculating the second activation from the input is:
$$\sigma(\begin{pmatrix}1 & _0a_{0}^0 & _0a_{1}^0\\\ 1 & _1a_{0}^0 & _0a_{1}^0\end{pmatrix} * \begin{pmatrix}b^0_0 & b^0_1 & b^0_2\\\ w^0_{0,0} & w^0_{0,1} & w^0_{0,2}\\\ w^0_{1,0} & w^0_{1,1} & w^0_{1,2}\end{pmatrix}) = \begin{pmatrix}_0a_{0}^1 & _0a_{1}^1 & _0a_{2}^1\\\ _1a_{0}^1 & _1a_{1}^1 & _1a_{2}^1\end{pmatrix}$$
The input of the first observation will be [0.8, 0.4], the second will be [0.3, 0.3] (they are completely random - just as the expected outputs).
So based on the above equation (sorry couldn't find other solutions to display it) $n^1$ (the neuron value, before the activation function) is:
+-----+-----+-----+
| 1 | 1 | 1 |
| 1 | 1 | 1 |
| 1 | 1 | 1 |
+---+-----+-----+-----+-----+-----+
| 1 0.8 0.4 | 2.2 | 2.2 | 2.2 |
| 1 0.3 0.3 | 1.6 | 1.6 | 1.6 |
+---+-----+-----+-----+-----+-----+
After taking the sigmoid, $a^1$ is:
[[0.9002495, 0.9002495, 0.9002495],
[0.8320184, 0.8320184, 0.8320184]]
This will get a "column" of 1s to get the bias added alone, will get matrix-multiplied with its weights ($w^1$) (containing the bias as well, in the top "row"), and "sigmoided", where we get $a^2$, and the same process continues until we get the output, $a^3$. They look like:
- $a^2$:
[[0.9758906, 0.9758906, 0.9758906],
[0.9705753, 0.9705753, 0.9705753]]
- $a^3$:
[[0.9806908, 0.9806908],
[0.9803864, 0.9803864]]
So far it all makes sense, as I initialised the weights with 1. Now the hard part. Calculate how much each weight affected the cost ($\sum C$), by following the chain rule: $$\frac{\partial\sum C}{\partial w^n} = \frac{\partial n^{n+1}}{\partial w^{n}}\frac{\partial a^{n+1}}{\partial n^{n+1}}\frac{\partial\sum C}{\partial a^{n+1}}$$
Where if $a^{n+1}$ in $\frac{\partial\sum C}{\partial a^{n+1}}$ is the output activation layer, because it has direct access to the expected output ($y$), it can be solved by (3 is just the index of the layer here as well):
$$\frac{\partial\sum C}{\partial a^{3}} = 2(a^{3} - y)$$
I don't care about splitting the cost, because the overall cost ($\sum C$) consists of $_0C_0$, that can only be affected by $_0a_0$, and so on, so for example
$$\frac{\partial\sum C}{\partial a^{3}_0} = \frac{\partial C_0}{\partial a^{3}_0} + \frac{\partial C_1}{\partial a^{3}_0} = 2(a^3_0 - y_0) + 0$$
Because our expected output is (random as well):
[[0.2, 0.4],
[1 , 0 ]]
It will end up in:
$$\begin{pmatrix}0.9806908 & 0.9806908\\\ 0.9803864 & 0.9803864\end{pmatrix} - \begin{pmatrix}0.2 & 0.4\\\ 1 & 0\end{pmatrix} = \begin{pmatrix}0.7806908 & 0.5806908\\\ -0.0196136 & 0.9803864\end{pmatrix} * 2 = \begin{pmatrix}1.5613816 & 1.1613816\\\ -0.0392272 & 1.9607728\end{pmatrix}$$.
Next step is $\frac{\partial a^{n+1}}{\partial n^{n+1}}$, where I get how $n$ affected $a$. Because $a$ came from applying the sigmoid function to $n$, and knowing that the derivative of the sigmoid function is:
$$\sigma (x)(1 - \sigma (x)) = (\sigma (x) - 1)\sigma (x) * -1$$
In our case is (the transformation is just to make it easier to work with it with TensorFlow):
$$\frac{\partial a^{3}}{\partial n^{3}} = (\sigma (n^{3}) - 1)\sigma (n^3) * -1 = (a^3 - 1)a^3 * -1$$
We can transform $\sigma (n^{3})$ to $a^3$ because that's exactly how we made it.
In our case it will end up in:
$$\begin{pmatrix}0.9806908 & 0.9806908\\\ 0.9803864 & 0.9803864\end{pmatrix} - 1 = \begin{pmatrix}-0.0193092 & -0.0193092\\\ -0.0196136 & -0.0196136\end{pmatrix} * \begin{pmatrix}0.9806908 & 0.9806908\\\ 0.9803864 & 0.9803864\end{pmatrix} * -1 = \begin{pmatrix}0.01893635 & 0.01893635\\\ 0.01922891 & 0.01922891\end{pmatrix}$$
Third step is $\frac{\partial n^{n+1}}{\partial w^{n}}$, which is the previous activation (we multiplied $a^n$ with $w^n$ to get $n^{n+1}$).
The goal is to find how much effect a weight layer had on the cost, so the shape of the result must match the weight layers shape. The shape of the weights is:
- it must have as much number of columns, as the next layers number of neurons
- it must have as much number of rows as the previous layers number of neurons plus 1 for bias
In the equation of:
$$\frac{\partial\sum C}{\partial w^n} = \frac{\partial n^{n+1}}{\partial w^{n}}\frac{\partial a^{n+1}}{\partial n^{n+1}}\frac{\partial\sum C}{\partial a^{n+1}}$$
the first part will give the "rows" of the weights, because it's related to the previous activations, and the second, and third part will give the "columns", because it's related to the current activations/neurons. They are connected through the weights, what we're investigating, how much costs they have, so with matrix multiplication, we will get back how the weights should be modified.
So first we do:
$$\frac{\partial a^{n+1}}{\partial n^{n+1}}\frac{\partial\sum C}{\partial a^{n+1}} = \begin{pmatrix}1.5613816 & 1.1613816\\\ -0.0392272 & 1.9607728\end{pmatrix} * \begin{pmatrix}0.01893635 & 0.01893635\\\ 0.01922891 & 0.01922891\end{pmatrix} = \begin{pmatrix}0.02956687 & 0.02199233\\\ -0.0007543 & 0.03770352\end{pmatrix}$$
So far, it's promising that all the values seem different.
For $\frac{\partial n^{n+1}}{\partial w^{n}}$ (previous activation layer) we have to add 1s (for biases), as an extra first column, and transpose it (in order to do the matrix multiplication).
And the problem:
+-------------+-------------+
| 0.02956687 | 0.02199233 |
| -0.0007543 | 0.03770352 |
+-----------+-----------+-------------+-------------+
| 1 1 | 0.0288126 | 0.0596958 |
| 0.9758906 0.9705753 | 0.028122 | 0.0580562 |
| 0.9758906 0.9705753 | 0.028122 | 0.0580562 |
| 0.9758906 0.9705753 | 0.028122 | 0.0580562 |
+-----------+-----------+-------------+-------------+
Because I've initialised the weights with the same values, as you can see above as well, all the activations (for each observation) are the same at each level. Not sure if the whole problem is caused by the same initial weights, but I'm expecting the network to adjust itself to the right direction, no matter what the previous weights are.
You could think that it's just the first layer, just the first iteration, but trust me. It goes like this over and over, no matter how much iterations. Probably this is the reason why it produces the output for the above example after 1000 iterations:
[[0.5992708, 0.1997655],
[0.5993174, 0.2004307]]
Expected:
[[0.2, 0.4],
[1 , 0 ]]
So. What am I doing wrong?
