When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization?

Question

I knew that Residual Network (ResNet) made He normal initialization popular. In ResNet, He normal initialization is used , while the first layer uses He uniform initialization.

I've looked through ResNet paper and "Delving Deep into Rectifiers" paper (He initialization paper), but I haven't found any mention on normal init vs uniform init.

Also:

Batch Normalization allows us to use much higher learning rates and be less careful about initialization.

In Batch Normalization paper's abstract, it is said that Batch Normalization allows us to be less careful about initialization.

ResNet itself is still care on when to use normal init vs uniform init (rather than just go with the uniform init).

So:

• When to use (He or Glorot) normal-distributed initialization over uniform initialization?
• What are normal-distributed initialization effects with Batch Normalization?

Notes aside:

• It rhymes to use normal init with Batch Normalization, but I haven't found any paper to back this fact.
• I knew that ResNet uses He init over Glorot init because He init does better on a deep network.
• I've understood about Glorot init vs He init.
• My question is about Normal vs Uniform init.

Answer 1

The normal vs uniform init seem to be rather unclear in fact.

If we refer solely on the Glorot's and He's initializations papers, they both use a similar theoritical analysis: they find a good variance for the distribution from which the initial parameters are drawn. This variance is adapted to the activation function used and is derived without explicitly considering the type of the distribution. As such, their theorical conclusions hold for any type of distribution of the determined variance. In fact, in the Glorot paper, a uniform distribution is used whereas in the He paper it is a gaussian one that is chosen. The only "explaination" given for this choice in the He paper is:

Recent deep CNNs are mostly initialized by random weights drawn from Gaussian distributions

with a reference to AlexNet paper. It was indeed released a little later than Glorot's initialization but however there is no justificaion in it of the use of a normal distribution.

In fact, in a discussion on Keras issues tracker, they also seem to be a little confused and basically it could only be a matter of preference... (i.e. hypotetically Bengio would prefer uniform distribution whereas Hinton would prefer normal ones...) One the discussion, there is a small benchmark comparing Glorot initialization using a uniform and a gaussian distribution. In the end, it seems that the uniform wins but it is not really clear.

In the original ResNet paper, it only says they used a gaussian He init for all the layers, I was not able to find where it is written that they used a uniform He init for the first layer. (maybe you could share a reference to this?)

As for the use of gaussian init with Batch Normalization, well, with BN the optimization process is less sensitive to initialization thus it is just a convention I would say.