My question would be about backpropagation and understanding the terms feedforward NN vs backpropagation. I have two questions really:

1. If I understand correctly, even a feedforward network updates its weights (via a delta rule for example). Isn't this also backpropagation? You have some random weights, run the data through the network, then cross-validate it with the desired output, then update the rules. This is backpropagation, right? So what's the difference between FFW NN and RNN? If you can't backpropagate on the other hand, how do you update the weights in a FFW NN?

2. I've seen NN architectures looking like this:

Basically all the neurons are being fed the same data, right? OK, the weights are randomized and thus different in the beginning, but how do you make sure Temp. Value #1 will be different from Temp. Value #2, if you use the same update rule?

Thank you!

I'm not an expert on the backpropagation algorithm, however I can explain something. Every neural network can update it's weights. It may do this in different ways, but it can. This is called backpropagation, regardless of the network architecture.

A feed forward network is a regular network, as seen in your picture. A value is received by a neuron, then passed on to the next one.

A recurrent neural network is almost the same as a FFN, the difference being that the RNN has some connections point 'backwards'. E.g. a neuron is connecteded to a neuron that has already done his 'job' during backpropagation. Because of this, the activations of the previous output have an effect on the new output.

On question #2

Interesting question. This has to do with weight initialization. Yes, you're right, each neuron in the hidden layer accepts the same connections. However, during the initalization process, they have received a random weight. Depending on your NN libary, the neurons might also have been initialized with a random bias.

So even though the same rule is applied, each neuron has different outcomes as all it's connections have different weights than the other neurons weights.

On your comment: just because all the neurons happen to have the same backpropagation function, doesn't mean they will end up with the same weights.

As they are initialized with random weights, each neurons error is different. Thus they have a different gradient, and will get new weights.

You also have to keep in mind that for a certain output to be reached, there are multiple solutions (due to non-linearity). So due to initialized random weights, one neuron might be close to a certain solution while another neuron is closer to the other.

Additionally, as was stated in the comments, a network works as a whole. The output neuron is also non-linear, and for most test cases, the output should be non-linear and the output neuron most likely requires that the hidden neurons activate at different input values.

• Thank you! I understand the first part completely now. But regarding the second, given the same input and the same goal, shouldn't all the weights converge after a number of iterations at the same value? As far as I know, gradient descent of the error function is often used to recalculate the weights, how does that not lead to the same weights? I'd think same input + random weights + same output + same weight-adjusting function = convergence to the same value over time, no matter the initial random weights. – lte__ Apr 19 '17 at 19:42
• @lte__ Your intuition for "same input + random weights + same output + same weight-adjusting function = convergence to the same value over time" is wrong. I am not clear on why exactly you think this. Maybe you are thinking about each neuron in isolation. But between layers there are lots of cross-links, so the neurons do not behave as if isolated and the network converges as a whole, not as a set of individual predictors. – Neil Slater Apr 19 '17 at 20:54
• @lte__ I understand your mindset, but just as Neil Slater said: a network converges as a whole. Read my updated answer. – Thomas W Apr 20 '17 at 8:57