What is the purpose of multiple neurons in a hidden layer?

Question

On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer.

Let me make the question more clear.

Take this classic image:

Clearly, the input layer is a vector with 3 components. Each of the three components is propagated to the hidden layer. Each neuron, in the hidden layer, sees the same vector with 3 components -- all neurons see the same data.

So we are at the hidden layer now. From what I read, this layer is normally just ReLus or sigmoids.

Correct me if I'm wrong, but a ReLu is a ReLu. Why would you need 4 of the exact same function, all seeing the exact same data?

What makes the red neurons in the hidden layer different from each other? Are they supposed to be different? I haven't read anything about tuning or setting parameters or perturbing different neurons to have them be different. But if they aren't different...then what's the point?

Text under the image above says, "A neural network is really just a composition of perceptrons, connected in different ways." They all look connected in the exact same way to me.

Answer 1

To explain using the sample neural network you have provided:

1. Purpose of the multiple inputs: Each input represents a feature of the input dataset.
2. Purpose of the hidden layer: Each neuron learns a different set of weights to represent different functions over the input data.
3. Purpose of the output layer: Each neuron represents a given class of the output (label/predicted variable).

If you used only a single neuron and no hidden layer, this network would only be able to learn linear decision boundaries. To learn non-linear decision boundaries when classifying the output, multiple neurons are required. By learning different functions approximating the output dataset, the hidden layers are able to reduce the dimensionality of the data as well as identify mode complex representations of the input data. If they all learned the same weights, they would be redundant and not useful.

The way they will learn different "weights" and hence different functions when fed the same data, is that when backpropagation is used to train the network, the errors represented by the output are different for each neuron. These errors are worked backwards to the hidden layer and then to the input layer to determine the most optimum value of weights that would minimize these errors.

This is why when implementing backpropagation algorithm, one of the most important steps is to randomly initialize the weights before starting the learning. If this is not done, then you would observe a large no. of neurons learning the exact same weights and give sub-optimal results.

---

Edited to answer additional questions:

• The only reason the neurons aren't redundant is because they've all been "trained" with different set of weights, hence, give a different output when presented with the same data. This is achieved by random initialization and back-propagation of errors.
• The outputs from the Orange neurons (use your diagram as an example), are "squashed" by each Blue neuron by applying the sigmoid or Relu function with the trained weights and the output of the orange neurons.

Answer 2

I think the key here is that when in training, having same weights (and same neurons in the hidden layer) will not lead to optimal solution, it is the different weights that will lead to lower difference between actual and predicted values.

I coded up this really dumb Neural Network (as a learning exercise). Maybe it could be of some help.

import numpy as np

class NeuralNetwork(object):
    def __init__(self, X, Y, hidden_layer_dim):
        self.X = X / np.max(X)
        self.Y = Y / np.max(Y) # Used for training
        self.hidden_layer_dim = hidden_layer_dim

    def initialize_weights(self):
        self.w1 = np.random.normal(0,1, (self.X.shape[1], self.hidden_layer_dim))
        self.w2 = np.random.normal(0,1, self.hidden_layer_dim)

    def forward(self, xi):
        """
        x1 is 2d array
        """
        # This method is also used for training 
        xi = xi / np.max(xi)
        z2 = np.dot(xi, self.w1)
        a2 = sigmoid(z2)
        z3 = np.dot(a2, self.w2)
        y_hat = sigmoid(z3)
        return y_hat

    def dump_train(self, n_iterations):
        min_mse = np.inf
        for i in range(n_iterations):
            w1 = np.random.normal(0,1, (self.X.shape[1], self.hidden_layer_dim))
            w2 = np.random.normal(0,1, self.hidden_layer_dim)

            z2 = np.dot(self.X, w1)
            a2 = sigmoid(z2)
            z3 = np.dot(a2, w2)
            y_hat = sigmoid(z3)

            diff = self.Y - y_hat
            mse = np.dot(diff, diff)

            if mse < min_mse:
                min_mse = mse
                print("min_mse: {}, iteration: {}".format(mse, i))
                self.w1 = w1
                self.w2 = w2

def sigmoid(a):
    return 1 / (1 + np.e ** (-a))

if __name__ == "__main__":
    my_x = np.array([[8,5], [7,5], [8,4],[8,1], [4, 3], [5,2], [4,2]], dtype=np.float)
    my_y = np.array([100, 90, 88, 60, 50, 45, 40], dtype=np.float)

    NN = NeuralNetwork(my_x, my_y, hidden_layer_dim=3)
    NN.initialize_weights()
    NN.dump_train(100000)

    new_x = [[8,4], [7,1], [3,3]]

    y_hat = NN.forward(new_x)

    print("prediction: {}".format(y_hat))
    print("weight 1: {}".format(NN.w1))
    print("weight 2: {}".format(NN.w2)) 

Result:

weight 1: [[-0.13787113 -1.30913914  0.64624687]
 [-1.76733779  0.77449265  1.61122177]]
weight 2: [-1.42489674 -1.94360005  2.56365303]

The weights are different.

Answer 3

I had the same doubt and I think this youtube video clears it. This video shows an example and while doing backpropagation, the weights are adjusted differently and thus instead of all the weights converging into the same value, it diverges.

https://www.youtube.com/watch?v=0e0z28wAWfg