Whats the advantage of He Intialization over Xavier Intialization?

Question

For Weights initialization, I read that He doesn't consider linear activation of neurons as Xavier Initialization; in this context, what does linear initialization mean?

Answer 1

Yeah, it is true actually that He doesn't consider linear activation of neurons as Xavier Initialization as He initialization is specifically designed to work well with ReLU activation functions, which are non-linear. Xavier initialization, on the other hand, is based on the assumption that the activation function used by the neuron is linear. Therefore, He initialization does not consider linear activation of neurons in the same way that Xavier initialization does. That means He initialization can be a good choice for initializing the weights of deep neural networks with ReLU activation functions.

And for your question regarding Linear initialization, it refers to initializing the weights of a neural network such that the output of each neuron is a linear function of its inputs.

Maybe you can give a try reading the following: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification