I'm trying to create a simple neural network with 2 input neurons, 1 hidden layer with 3 neurons and an output layer with single neuron and to train it for XOR problem.
As per my understanding of neural nets, I'm planning to have in each neuron a weighted linear function of inputs and a Sigmoid activation function. So for example, the first input neuron with input
x will compute
a being a parameter and
b being its bias. Then it will feed
c to Sigmoid function to compute
z=1/(1+exp(-c)) which will be its output. Please correct me if I'm wrong in any of this.
Now, I'm trying to follow this article to create back-propagation differential equations. It says that
dc = z * (1-z) * dz where dz is the gradient of the green edge and then
da = x * dc where
da is the gradient computed in this iteration of backpropagation.
My question is that there are three
zs (one corresponding to each outward edge) and their value(weight) will be identical (
z=1/(1+exp(-c))). And I assume, then all values and gradients in the three neurons in hidden layer will also be identical. Then what's the point of having three neurons in hidden layer? What causes the randomness which can lead to three different gradients across each of the three green edges? And if they actually are different, how can I compute
dc which is simply
z * (1-z) * dz but there are three
zs and three