Setting the initial weights as all zeros will have the output dependent on the bias and setting the weights of all the neurons of a layer as same, will update the gradients in same way thus removing the effect of non-linearity and under fitting of model.
Now I want to ask that if a model has 3 layers (of course with same number of neurons) and we initialise all the weights and biases of the network as same, given no two neurons in the same layer has same values. For example, let us suppose that weights of all the
W in all the layer at the beginning is as:
Now what will happen?
I guess the model will be under fitting and will be linear in nature because in case we use
ReLu, the output will be just the scaled version of the prevision layer?