When researching online, I keep finding that Xavier/Glorot initialization is:

enter image description here enter image description here

however, the original paper by Glorot said that this was a common initialization strategy that they soon found did not work great:

enter image description here

and instead they came up with a new initialization scheme:

enter image description here

To me, these look like two completely different schemes. The first only is only a function of the number of inputs to a layer, and the second is a function of both the number of inputs AND the number of outputs. I don't know if the normal vs uniform distribution makes up for the difference.

So my questions are:

  • which one is the ACTUAL Xavier initialization?
  • what are the differences between the two schemes?
  • which one do I use?
  • what did not work about the first scheme that prompted Glorot to come up with the second one?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.