Skip to main content

Questions tagged [glorot-initialization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
5 votes
1 answer
10k views

How/What to initialize the hidden states in RNN sequence-to-sequence models?

In an RNN sequence-to-sequence model, the encode input hidden states and the output's hidden states needs to be initialized before training. What values should we initialize them with? How should we ...
alvas's user avatar
  • 2,410
4 votes
1 answer
5k views

What is glorot uniform?

While reading Bengio and Glorot, came across this, but after going through paper, intution about the algorithm was not so deep, care to explain why was this needed and where they needed to implement ...
TheExorcist's user avatar
2 votes
0 answers
95 views

TF: What is the difference between the 'kernel weights' and the 'recurrent kernel weights' in LSTMs/GRUs?

Context: I am trying to understand the differences between the GRU/LSTM cells from tensorflow and pytorch (for research reproducibility) and noticed that TensorFlow differentiates between the ...
Robin van Hoorn's user avatar
1 vote
0 answers
15 views

confusion with Xavier Initiliazation definition

When researching online, I keep finding that Xavier/Glorot initialization is: however, the original paper by Glorot said that this was a common initialization strategy that they soon found did not ...
tom394's user avatar
  • 11
0 votes
0 answers
6 views

How to derive the formula 13 in the Xavier Initialization paper

How to derive the formula 13 in the Xavier Initialization paper Understanding the difficulty of training deep feedforward neural networks from the formula 6?
mon's user avatar
  • 731
0 votes
0 answers
23 views

Clarifying the arguments of "Understanding the difficulty of training deep feedforward neural networks"

I made the decision to try to push through the paper "Understanding the difficulty of training deep feedforward neural networks". (The paper is given as a reference in "Hands-On Machine ...
Chris's user avatar
  • 101
0 votes
1 answer
33 views

How is the property in eq 15 obtained for Xavier initialization

I am new in this field so please be gentle with terminology. In the original paper; "Understanding the difficulty of training deep feedforward neural networks", I dont understand how equation 15 is ...
Laughing Man's user avatar