I want to create a 3 layers neural network from scratch to perform linear regression. The first and the second layer have 2 neurons, and the last layer has one neuron.
Feature vector x is divided into $x_{1}, x_{2}$ where $x_{1} = ax, x_2 = (1-a)x, 0 < a < 1$
Hence, there are 6 weights: $w_{111}, w_{121}, w_{211}, w_{221}, w_{112}, w_{212}$.
To note that I'm not including biases since they aren't the object of the question and the activation function is linear, so I'm not including that as well.
Reading the answer to this question, I'm computing the gradient of each weight in this way: ($w_{111}$ and $w_{121}$ are took as examples)
$\frac{∂C}{w_{111}} = \frac{1}{m} \sum_{i=1}^m \frac{∂L_{i}}{w_{111}}$,
$\frac{∂L_{i}}{w_{111}} = (y-y')\frac{∂y'}{∂w_{111}}$,
$\frac{∂y'}{∂w_{111}} = \frac{∂z_{a1}}{∂w_{111}}$,
$\frac{∂z_{a1}}{∂w_{111}} = \frac{∂}{∂w_{111}}(w_{111}x_{1} + w_{121}x_{2}) = x_{1}$
$\frac{∂C}{w_{111}} = \frac{1}{m} \sum_{i=1}^m (y-y')x_{i1}$
$\frac{∂C}{w_{112}} = \frac{1}{m} \sum_{i=1}^m (y-y')x_{i2}$
But, since $z_{a1} = w_{111}x_{1} + w_{121}x_{2}$, and $z_{a2} = w_{211}x_{1} + w_{221}x_{2}$,
isn't $\frac{∂C}{w_{111}} = \frac{1}{m} \sum_{i=1}^m (y-y')x_{i1} = \frac{∂C}{w_{211}}$, and so, isn't $w_{111} = w_{211}$, and $w_{121} = w_{221}$?