On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer.

Let me make the question more clear.

Take this classic image:

enter image description here

Clearly, the input layer is a vector with 3 components. Each of the three components is propagated to the hidden layer. Each neuron, in the hidden layer, sees the same vector with 3 components -- all neurons see the same data.

So we are at the hidden layer now. From what I read, this layer is normally just ReLus or sigmoids.

Correct me if I'm wrong, but a ReLu is a ReLu. Why would you need 4 of the exact same function, all seeing the exact same data?

What makes the red neurons in the hidden layer different from each other? Are they supposed to be different? I haven't read anything about tuning or setting parameters or perturbing different neurons to have them be different. But if they aren't different...then what's the point?

Text under the image above says, "A neural network is really just a composition of perceptrons, connected in different ways." They all look connected in the exact same way to me.

  • $\begingroup$ This online book has javascript illustrations that allow you to change weights and understand how the hidden layer allows computing any function: neuralnetworksanddeeplearning.com/chap4.html. That being said, keep in mind nobody knows why neural networks work so well. They are basically logistic regressions on steroids. $\endgroup$ Commented Sep 23, 2016 at 9:37
  • $\begingroup$ Careful with that quote. The classic perceptron's activation function was either on/off. the insight that let us have backprop and thus multi layer perceptrons was seeing that we could differentiate the error (and thus be able to see what direction is "downhill") if we used sigmoid (or really any differentiable function). $\endgroup$ Commented Mar 3, 2019 at 23:09

3 Answers 3


To explain using the sample neural network you have provided:

  1. Purpose of the multiple inputs: Each input represents a feature of the input dataset.
  2. Purpose of the hidden layer: Each neuron learns a different set of weights to represent different functions over the input data.
  3. Purpose of the output layer: Each neuron represents a given class of the output (label/predicted variable).

If you used only a single neuron and no hidden layer, this network would only be able to learn linear decision boundaries. To learn non-linear decision boundaries when classifying the output, multiple neurons are required. By learning different functions approximating the output dataset, the hidden layers are able to reduce the dimensionality of the data as well as identify mode complex representations of the input data. If they all learned the same weights, they would be redundant and not useful.

The way they will learn different "weights" and hence different functions when fed the same data, is that when backpropagation is used to train the network, the errors represented by the output are different for each neuron. These errors are worked backwards to the hidden layer and then to the input layer to determine the most optimum value of weights that would minimize these errors.

This is why when implementing backpropagation algorithm, one of the most important steps is to randomly initialize the weights before starting the learning. If this is not done, then you would observe a large no. of neurons learning the exact same weights and give sub-optimal results.

Edited to answer additional questions:

  • The only reason the neurons aren't redundant is because they've all been "trained" with different set of weights, hence, give a different output when presented with the same data. This is achieved by random initialization and back-propagation of errors.
  • The outputs from the Orange neurons (use your diagram as an example), are "squashed" by each Blue neuron by applying the sigmoid or Relu function with the trained weights and the output of the orange neurons.
  • $\begingroup$ Ahah! So...am I correct to say that each neuron sees the exact same input data? And that the only reason the neurons aren't redundant is because they've been initialized with different random values? $\endgroup$ Commented Sep 16, 2016 at 3:10
  • $\begingroup$ And how do the outputs from the hidden layer combine to point towards a class? Average? Majority rule? $\endgroup$ Commented Sep 16, 2016 at 3:22
  • $\begingroup$ Not sure I totally understand how the 'squashing' happens at that last step...but I'm happy to accept your answer regardless. $\endgroup$ Commented Sep 16, 2016 at 4:42
  • 1
    $\begingroup$ By Squashing I mean applying the sigmoid function to reduce all inputs to a neuron into a single output i.e. yi = sigmoid( wi.xi + b); where xi are inputs, bi are biases and yi are outputs of a layer. $\endgroup$ Commented Sep 16, 2016 at 4:59
  • $\begingroup$ All 4 Red neurons are sent into a softmax sigmoid then, to get the final 2 outputs? $\endgroup$ Commented Sep 16, 2016 at 5:23

I think the key here is that when in training, having same weights (and same neurons in the hidden layer) will not lead to optimal solution, it is the different weights that will lead to lower difference between actual and predicted values.

I coded up this really dumb Neural Network (as a learning exercise). Maybe it could be of some help.

import numpy as np

class NeuralNetwork(object):
    def __init__(self, X, Y, hidden_layer_dim):
        self.X = X / np.max(X)
        self.Y = Y / np.max(Y) # Used for training
        self.hidden_layer_dim = hidden_layer_dim

    def initialize_weights(self):
        self.w1 = np.random.normal(0,1, (self.X.shape[1], self.hidden_layer_dim))
        self.w2 = np.random.normal(0,1, self.hidden_layer_dim)

    def forward(self, xi):
        x1 is 2d array
        # This method is also used for training 
        xi = xi / np.max(xi)
        z2 = np.dot(xi, self.w1)
        a2 = sigmoid(z2)
        z3 = np.dot(a2, self.w2)
        y_hat = sigmoid(z3)
        return y_hat

    def dump_train(self, n_iterations):
        min_mse = np.inf
        for i in range(n_iterations):
            w1 = np.random.normal(0,1, (self.X.shape[1], self.hidden_layer_dim))
            w2 = np.random.normal(0,1, self.hidden_layer_dim)

            z2 = np.dot(self.X, w1)
            a2 = sigmoid(z2)
            z3 = np.dot(a2, w2)
            y_hat = sigmoid(z3)

            diff = self.Y - y_hat
            mse = np.dot(diff, diff)

            if mse < min_mse:
                min_mse = mse
                print("min_mse: {}, iteration: {}".format(mse, i))
                self.w1 = w1
                self.w2 = w2

def sigmoid(a):
    return 1 / (1 + np.e ** (-a))

if __name__ == "__main__":
    my_x = np.array([[8,5], [7,5], [8,4],[8,1], [4, 3], [5,2], [4,2]], dtype=np.float)
    my_y = np.array([100, 90, 88, 60, 50, 45, 40], dtype=np.float)

    NN = NeuralNetwork(my_x, my_y, hidden_layer_dim=3)

    new_x = [[8,4], [7,1], [3,3]]

    y_hat = NN.forward(new_x)

    print("prediction: {}".format(y_hat))
    print("weight 1: {}".format(NN.w1))
    print("weight 2: {}".format(NN.w2)) 


weight 1: [[-0.13787113 -1.30913914  0.64624687]
 [-1.76733779  0.77449265  1.61122177]]
weight 2: [-1.42489674 -1.94360005  2.56365303]

The weights are different.


I had the same doubt and I think this youtube video clears it. This video shows an example and while doing backpropagation, the weights are adjusted differently and thus instead of all the weights converging into the same value, it diverges.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.