5
$\begingroup$

In an RNN sequence-to-sequence model, the encode input hidden states and the output's hidden states needs to be initialized before training.

What values should we initialize them with? How should we initialize them?

From the PyTorch tutorial, it simply initializes zeros to the hidden states.

Is initializing zero the usual way of initializing hidden states in RNN seq2seq networks?

How about glorot initialization?

For a single-layer vanilla RNN wouldn't the fan-in and fan-out be equals to $(1 + 1)$ which gives a variance of $1$ and the gaussian distribution with $mean=0$ gives us a uniform distribution of $0$s.

for-each input-hidden weight
  variance = 2.0 / (fan-in +fan-out)
  stddev = sqrt(variance)
  weight = gaussian(mean=0.0, stddev)
end-for

For single layer encoder-decoder architecture with attention, if we use glorot, we'll get a very very small variance when initializing the decoder hidden state since the fan-in would include the attention which is mapped to all possible vocabulary from the encoder output. So we result in a gaussian mean of ~= 0 too since stdev is really really small.

What other initialization methods are there, esp. for the use on RNN seq2seq models?

$\endgroup$

1 Answer 1

5
$\begingroup$

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural networks.

Since your question is asking about hidden state initialization: Hidden states on the other hand can be initialized in a variety of ways, initializing to zero is indeed common. Other methods include sampling from Gaussian or other distributions. In relation to RNN's this defines what a RNN starts with as its 'memory'. Two common approaches seem to be either a noisy initialization (from some sort of distribution or a random number generator), or a learned initialization.

To synthesize the link above; initializing hidden states with zeros can lead to the network learning to adapt from a zero hidden state, rather than minimizing the loss for a long sequence (it follows that this is more of a problem for short sequences). If there are enough sequences it can make sense to have the initial state be a trained variable that is a function of the error during back propagation.

$\endgroup$
1
  • $\begingroup$ Do you have any reference or example that does what you mention If there are enough sequences it can make sense to have the initial state be a trained variable that is a function of the error during back propagation. ? Maybe by training it during pauses / freezes of all other parameters in the seq2seq ? $\endgroup$
    – Blupon
    Sep 2, 2022 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.