When researching online, I keep finding that Xavier/Glorot initialization is:
however, the original paper by Glorot said that this was a common initialization strategy that they soon found did not work great:
and instead they came up with a new initialization scheme:
To me, these look like two completely different schemes. The first only is only a function of the number of inputs to a layer, and the second is a function of both the number of inputs AND the number of outputs. I don't know if the normal vs uniform distribution makes up for the difference.
So my questions are:
- which one is the ACTUAL Xavier initialization?
- what are the differences between the two schemes?
- which one do I use?
- what did not work about the first scheme that prompted Glorot to come up with the second one?