1
$\begingroup$

I'm trying to create a simple neural network with 2 input neurons, 1 hidden layer with 3 neurons and an output layer with single neuron and to train it for XOR problem. Network structure As per my understanding of neural nets, I'm planning to have in each neuron a weighted linear function of inputs and a Sigmoid activation function. So for example, the first input neuron with input x will compute c=ax+b, a being a parameter and b being its bias. Then it will feed c to Sigmoid function to compute z=1/(1+exp(-c)) which will be its output. Please correct me if I'm wrong in any of this.

Now, I'm trying to follow this article to create back-propagation differential equations. It says that dc = z * (1-z) * dz where dz is the gradient of the green edge and then da = x * dc where da is the gradient computed in this iteration of backpropagation.

My question is that there are three zs (one corresponding to each outward edge) and their value(weight) will be identical (z=1/(1+exp(-c))). And I assume, then all values and gradients in the three neurons in hidden layer will also be identical. Then what's the point of having three neurons in hidden layer? What causes the randomness which can lead to three different gradients across each of the three green edges? And if they actually are different, how can I compute dc which is simply z * (1-z) * dz but there are three zs and three dzs?

$\endgroup$
1
$\begingroup$

The gradient is actually calculated w.t.r to the weights so that each neurone would have a different 'change in the weight' according to its previous weight. In other words, not all the 3 neurones in the hidden layer will be penalized with the same value because the initial weights at t=0 are different for these neurones.

It is well explained here: https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

I hope this short answer helped!

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.