From the lstm equations, e.g. as they appear on p406 of the Deep Learning Book, it looks to me like initializing with zeros (as is common practice), must always produce a strictly positive output. When $s_0 = 0$, then the $i$th output unit at the first time step can be written as $h_i = \sigma(A)tanh((\sigma(B)(\sigma(C))$ where $A$, $B$ and $C$ are linear functions on the current input and the previous output. But isn't then $h_i > 0$ for all real numbers $A$, $B$ and $C$?


Cannot find the equations in your reference, so I take them from wikipedia. At first time step the following equation holds: $$h_{i} = \sigma(A)\tanh((\sigma(B)(\tanh(C))$$ where $A, B$ and $C$ are values calculated from linear functions. So it can be negative if $C$ is so.

  • $\begingroup$ Hmm, yes you're right about the Wikipedia equations. What is the problem with my link, may I ask? because the issue must be that I am misreading the equations therein, or there is perhaps an error in the book. They are on page 406. In particular, equation (10.41) seems to use the sigmoid activation function for the internal state update, where Wikipedia (and a few other online sources I just checked) uses tanh. $\endgroup$
    – ludog
    Apr 8 '19 at 20:14
  • $\begingroup$ @ludog oh! I didn't read the "on p406".. sorry. Yes after looking at the book's equations it seems It doesn't use $\tanh$ in the state update... emmm... Well, maybe it is a miss typing error. I am no one to fully confirm that the book is wrong though $\endgroup$
    – lsmor
    Apr 11 '19 at 7:39
  • 1
    $\begingroup$ Seems to me it must be a typo. In any case, thanks for pointing this out. :) $\endgroup$
    – ludog
    Apr 11 '19 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.