I'm following a couple of tutorials on RNNs and the instructor said that we should always set the initial hidden state in our RNNs to a tensor of all zeros and I couldn't really understand why. Even when I looked online, it said that fixing the hidden state to zero is a common choice and that this has been shown to force the RNN to focus on the input sequence regardless of the initial choice of the hidden state. What do they mean by that exactly?

  • $\begingroup$ setting initial hidden states to random values roughly means the RNN is at a random state in time and not in an initial state (where previous hidden states would be zero), right? $\endgroup$ – Nikos M. Apr 3 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.