Does every layer of a Neural Network require weight initialization or just the first? Does the first layer feed into the next layer and initialize itself? My intuition is that every layer needs its own initialization but I'm finding it hard to see that said explicitly. Thanks!


Yes, every layer that contains weights needs to be initialised.

Every weight is just a number (usually a decimal, floating point number). It has to be initialised with some value so that the algorithm (e.g. backpropagation) has some values to work with.

Any learned parameters need to be initialised, as they will be adapted during training. This means you can inialised e.g. the parameters of momentum, and it is changed over time.

There are "layers" that don't need initialisation. I mean layers that perform some operations, but don't have dynamic weights; weights that are altered during training.

An example would be a standard dropout layer (with no learned parameters). This is called a layer, but doesn't require initialisation as it doesn't have any weights that are changed during training.

  • $\begingroup$ Thanks for the nuance there! In the output layer, I assume you also need to initialize the weights, yes? Is there a special strategy for this particular initialization (say when using a sigmoid activation function) ? $\endgroup$ – Ciaran Kelly Feb 13 '19 at 14:45
  • $\begingroup$ @CiaranKelly - There are several well-known initialisation methods in general - key types are: zero, random, He and Xavier. Most deep learning frameworks offer all the above. As far as I know, there isn't an initialisation method known to be particularly fitting for a final linear layer. $\endgroup$ – n1k31t4 Feb 13 '19 at 16:12

The weights of every single layer need to be initialized. I think you are confusing between the input and the weights of the network.

During the forward pass, the input passes thru the first layer, gets transformed by the layer's weights (usually convolved or multiplied by them), and then the output of the layer is passed as input to the next layer, where again it is transformed by its weights (and so on and on). So before the initial first forward pass, all weights must be initialized.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.