Are some weight gradients equal?

Question

I want to create a 3 layers neural network from scratch to perform linear regression. The first and the second layer have 2 neurons, and the last layer has one neuron.

Feature vector x is divided into $x_{1}, x_{2}$ where $x_{1} = ax, x_2 = (1-a)x, 0 < a < 1$

Hence, there are 6 weights: $w_{111}, w_{121}, w_{211}, w_{221}, w_{112}, w_{212}$.

To note that I'm not including biases since they aren't the object of the question and the activation function is linear, so I'm not including that as well.

Reading the answer to this question, I'm computing the gradient of each weight in this way: ($w_{111}$ and $w_{121}$ are took as examples)

$\frac{∂C}{w_{111}} = \frac{1}{m} \sum_{i=1}^m \frac{∂L_{i}}{w_{111}}$,
$\frac{∂L_{i}}{w_{111}} = (y-y')\frac{∂y'}{∂w_{111}}$,
$\frac{∂y'}{∂w_{111}} = \frac{∂z_{a1}}{∂w_{111}}$,
$\frac{∂z_{a1}}{∂w_{111}} = \frac{∂}{∂w_{111}}(w_{111}x_{1} + w_{121}x_{2}) = x_{1}$

$\frac{∂C}{w_{111}} = \frac{1}{m} \sum_{i=1}^m (y-y')x_{i1}$
$\frac{∂C}{w_{112}} = \frac{1}{m} \sum_{i=1}^m (y-y')x_{i2}$

But, since $z_{a1} = w_{111}x_{1} + w_{121}x_{2}$, and $z_{a2} = w_{211}x_{1} + w_{221}x_{2}$,

isn't $\frac{∂C}{w_{111}} = \frac{1}{m} \sum_{i=1}^m (y-y')x_{i1} = \frac{∂C}{w_{211}}$, and so, isn't $w_{111} = w_{211}$, and $w_{121} = w_{221}$?

Answer 1

The problem was that $\frac{∂y'}{∂w_{111}} \neq \frac{∂z_{a1}}{∂w_{111}}$, but $\frac{∂y'}{∂w_{111}} = \frac{∂z_{b}}{∂w_{111}}$, where $z_b$ is the sum of the products of all the z-vectors of the last layer with the weights.

So, here is an example of a NN with 3 layers, where the first two have 2 neurons and the last one has one neuron:

(with no activation function, layer n.1 z-vectors are just $x_i$)

• layer n. 2

$z_{11} = w_{111}x_1 + w_{112}x_2$
$z_{12} = w_{121}x_1 + w_{122}x_2$

• layer n. 3

$z_{21} = z_b = w_{211}z_{11} + w_{212}z_{12} $, hence:

$z_{21} = w_{211}(w_{111}x_1 + w_{112}x_2) + w_{212}(w_{121}x_1 + w_{122}x_2)$

Now, let's compute the derivatives:

• weights of layer n. 2

$\frac{∂z_b}{∂w_{111}} = x_1w_{211}$
$\frac{∂z_b}{∂w_{121}} = x_1w_{212}$
$\frac{∂z_b}{∂w_{112}} = x_2w_{211}$
$\frac{∂z_b}{∂w_{122}} = x_2w_{212}$

• weights of layer n. 3

$\frac{∂z_b}{∂w_{211}} = z_{11}$
$\frac{∂z_b}{∂w_{212}} = z_{12}$

As you can see, there is no weight derivative equal to another one.

I hope this can be useful to anyone interested in studying backpropagation.