On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer.
Let me make the question more clear.
Take this classic image:
Clearly, the input layer is a vector with 3 components. Each of the three components is propagated to the hidden layer. Each neuron, in the hidden layer, sees the same vector with 3 components -- all neurons see the same data.
So we are at the hidden layer now. From what I read, this layer is normally just ReLus or sigmoids.
Correct me if I'm wrong, but a ReLu is a ReLu. Why would you need 4 of the exact same function, all seeing the exact same data?
What makes the red neurons in the hidden layer different from each other? Are they supposed to be different? I haven't read anything about tuning or setting parameters or perturbing different neurons to have them be different. But if they aren't different...then what's the point?
Text under the image above says, "A neural network is really just a composition of perceptrons, connected in different ways." They all look connected in the exact same way to me.