# What is the purpose of multiple neurons in a hidden layer?

On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer.

Let me make the question more clear.

Take this classic image:

Clearly, the input layer is a vector with 3 components. Each of the three components is propagated to the hidden layer. Each neuron, in the hidden layer, sees the same vector with 3 components -- all neurons see the same data.

So we are at the hidden layer now. From what I read, this layer is normally just ReLus or sigmoids.

Correct me if I'm wrong, but a ReLu is a ReLu. Why would you need 4 of the exact same function, all seeing the exact same data?

What makes the red neurons in the hidden layer different from each other? Are they supposed to be different? I haven't read anything about tuning or setting parameters or perturbing different neurons to have them be different. But if they aren't different...then what's the point?

Text under the image above says, "A neural network is really just a composition of perceptrons, connected in different ways." They all look connected in the exact same way to me.

• This online book has javascript illustrations that allow you to change weights and understand how the hidden layer allows computing any function: neuralnetworksanddeeplearning.com/chap4.html. That being said, keep in mind nobody knows why neural networks work so well. They are basically logistic regressions on steroids. – Ricardo Cruz Sep 23 '16 at 9:37
• Careful with that quote. The classic perceptron's activation function was either on/off. the insight that let us have backprop and thus multi layer perceptrons was seeing that we could differentiate the error (and thus be able to see what direction is "downhill") if we used sigmoid (or really any differentiable function). – TheGrimmScientist Mar 3 '19 at 23:09

To explain using the sample neural network you have provided:

1. Purpose of the multiple inputs: Each input represents a feature of the input dataset.
2. Purpose of the hidden layer: Each neuron learns a different set of weights to represent different functions over the input data.
3. Purpose of the output layer: Each neuron represents a given class of the output (label/predicted variable).

If you used only a single neuron and no hidden layer, this network would only be able to learn linear decision boundaries. To learn non-linear decision boundaries when classifying the output, multiple neurons are required. By learning different functions approximating the output dataset, the hidden layers are able to reduce the dimensionality of the data as well as identify mode complex representations of the input data. If they all learned the same weights, they would be redundant and not useful.

The way they will learn different "weights" and hence different functions when fed the same data, is that when backpropagation is used to train the network, the errors represented by the output are different for each neuron. These errors are worked backwards to the hidden layer and then to the input layer to determine the most optimum value of weights that would minimize these errors.

This is why when implementing backpropagation algorithm, one of the most important steps is to randomly initialize the weights before starting the learning. If this is not done, then you would observe a large no. of neurons learning the exact same weights and give sub-optimal results.

• The only reason the neurons aren't redundant is because they've all been "trained" with different set of weights, hence, give a different output when presented with the same data. This is achieved by random initialization and back-propagation of errors.
• The outputs from the Orange neurons (use your diagram as an example), are "squashed" by each Blue neuron by applying the sigmoid or Relu function with the trained weights and the output of the orange neurons.
• Ahah! So...am I correct to say that each neuron sees the exact same input data? And that the only reason the neurons aren't redundant is because they've been initialized with different random values? – Monica Heddneck Sep 16 '16 at 3:10
• And how do the outputs from the hidden layer combine to point towards a class? Average? Majority rule? – Monica Heddneck Sep 16 '16 at 3:22
• Not sure I totally understand how the 'squashing' happens at that last step...but I'm happy to accept your answer regardless. – Monica Heddneck Sep 16 '16 at 4:42
• By Squashing I mean applying the sigmoid function to reduce all inputs to a neuron into a single output i.e. yi = sigmoid( wi.xi + b); where xi are inputs, bi are biases and yi are outputs of a layer. – Sandeep S. Sandhu Sep 16 '16 at 4:59
• All 4 Red neurons are sent into a softmax sigmoid then, to get the final 2 outputs? – Monica Heddneck Sep 16 '16 at 5:23

I think the key here is that when in training, having same weights (and same neurons in the hidden layer) will not lead to optimal solution, it is the different weights that will lead to lower difference between actual and predicted values.

I coded up this really dumb Neural Network (as a learning exercise). Maybe it could be of some help.

import numpy as np

class NeuralNetwork(object):
def __init__(self, X, Y, hidden_layer_dim):
self.X = X / np.max(X)
self.Y = Y / np.max(Y) # Used for training
self.hidden_layer_dim = hidden_layer_dim

def initialize_weights(self):
self.w1 = np.random.normal(0,1, (self.X.shape[1], self.hidden_layer_dim))
self.w2 = np.random.normal(0,1, self.hidden_layer_dim)

def forward(self, xi):
"""
x1 is 2d array
"""
# This method is also used for training
xi = xi / np.max(xi)
z2 = np.dot(xi, self.w1)
a2 = sigmoid(z2)
z3 = np.dot(a2, self.w2)
y_hat = sigmoid(z3)
return y_hat

def dump_train(self, n_iterations):
min_mse = np.inf
for i in range(n_iterations):
w1 = np.random.normal(0,1, (self.X.shape[1], self.hidden_layer_dim))
w2 = np.random.normal(0,1, self.hidden_layer_dim)

z2 = np.dot(self.X, w1)
a2 = sigmoid(z2)
z3 = np.dot(a2, w2)
y_hat = sigmoid(z3)

diff = self.Y - y_hat
mse = np.dot(diff, diff)

if mse < min_mse:
min_mse = mse
print("min_mse: {}, iteration: {}".format(mse, i))
self.w1 = w1
self.w2 = w2

def sigmoid(a):
return 1 / (1 + np.e ** (-a))

if __name__ == "__main__":
my_x = np.array([[8,5], [7,5], [8,4],[8,1], [4, 3], [5,2], [4,2]], dtype=np.float)
my_y = np.array([100, 90, 88, 60, 50, 45, 40], dtype=np.float)

NN = NeuralNetwork(my_x, my_y, hidden_layer_dim=3)
NN.initialize_weights()
NN.dump_train(100000)

new_x = [[8,4], [7,1], [3,3]]

y_hat = NN.forward(new_x)

print("prediction: {}".format(y_hat))
print("weight 1: {}".format(NN.w1))
print("weight 2: {}".format(NN.w2))


Result:

weight 1: [[-0.13787113 -1.30913914  0.64624687]
[-1.76733779  0.77449265  1.61122177]]
weight 2: [-1.42489674 -1.94360005  2.56365303]


The weights are different.