0
$\begingroup$

enter image description here

This is a GRU. Now, what will be the value of $h_t$, at $t$=$0$. That is, what will be the value of the hidden state at just the starting?

$\endgroup$
0
$\begingroup$

If you think intuitively what h_t (that is h_t-1) shoud represent is, very freely speaking, amount of rembering from the previous steps. Since you are just starting, and want to remember everything, very intuitive thing is to set the value at 1. Than you can see in the equations that the mutiplication with h_t-1 makes no difference in the beginning.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. Can't we initialize with 0? Would it make the network worse? Because I saw somebody initializing with zeros on the internet. $\endgroup$
    – Batman
    Nov 1 '20 at 18:02
  • $\begingroup$ hmmm well if you insert it in equation terms will dissapear, you see that right? $\endgroup$
    – Noah Weber
    Nov 1 '20 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.