0
$\begingroup$

For Weights initialization, I read that He doesn't consider linear activation of neurons as Xavier Initialization; in this context, what does linear initialization mean?

$\endgroup$

1 Answer 1

1
$\begingroup$

Yeah, it is true actually that He doesn't consider linear activation of neurons as Xavier Initialization as He initialization is specifically designed to work well with ReLU activation functions, which are non-linear. Xavier initialization, on the other hand, is based on the assumption that the activation function used by the neuron is linear. Therefore, He initialization does not consider linear activation of neurons in the same way that Xavier initialization does. That means He initialization can be a good choice for initializing the weights of deep neural networks with ReLU activation functions.

And for your question regarding Linear initialization, it refers to initializing the weights of a neural network such that the output of each neuron is a linear function of its inputs.

Maybe you can give a try reading the following: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

$\endgroup$
1
  • $\begingroup$ Thanks for the answer; now it makes sense. $\endgroup$
    – Carpediem
    Dec 17, 2022 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.