What what will happen if all the layers of a MLP or any DL architecture are set as same in the beginning?

Question

Setting the initial weights as all zeros will have the output dependent on the bias and setting the weights of all the neurons of a layer as same, will update the gradients in same way thus removing the effect of non-linearity and under fitting of model.

Now I want to ask that if a model has 3 layers (of course with same number of neurons) and we initialise all the weights and biases of the network as same, given no two neurons in the same layer has same values. For example, let us suppose that weights of all the W in all the layer at the beginning is as:

[[0.1,0.2,0.3],
 [0.4,0.5,0.6]]

Now what will happen?

I guess the model will be under fitting and will be linear in nature because in case we use ReLu, the output will be just the scaled version of the prevision layer?

Answer 1

There is no correlation between underfitting/overfitting with the initialization of the weights.

The problem with initialization of the weight is related to 2 main things:

1. Back propagation: if you initialize the weight to the same value, all the weight concur to the current "error"/"loss" in the same proportion, therefore will have the same update, which causes the NN to learn almost nothing
2. Saturation of activations: for example, suppose you initialize all the weights to be negative, and your dataset has MinMax scaled features in [0,1], if you do the product you will have a vector of negative values, and if you use ReLU as activation, those will be mapped to 0, and therefore also the gradient will be zero, leading to no learning at all

There are many other reasons behind the importance of the weights initialization, but there are plenty of good solutions already implemented that solves this problem (it's a bit complicated because the distribution of the initialization depends on the activation that you are using)