# how can LSTM output ever be negative with 0 initialization?

From the lstm equations, e.g. as they appear on p406 of the Deep Learning Book, it looks to me like initializing with zeros (as is common practice), must always produce a strictly positive output. When $$s_0 = 0$$, then the $$i$$th output unit at the first time step can be written as $$h_i = \sigma(A)tanh((\sigma(B)(\sigma(C))$$ where $$A$$, $$B$$ and $$C$$ are linear functions on the current input and the previous output. But isn't then $$h_i > 0$$ for all real numbers $$A$$, $$B$$ and $$C$$?

Cannot find the equations in your reference, so I take them from wikipedia. At first time step the following equation holds: $$h_{i} = \sigma(A)\tanh((\sigma(B)(\tanh(C))$$ where $$A, B$$ and $$C$$ are values calculated from linear functions. So it can be negative if $$C$$ is so.
• @ludog oh! I didn't read the "on p406".. sorry. Yes after looking at the book's equations it seems It doesn't use $\tanh$ in the state update... emmm... Well, maybe it is a miss typing error. I am no one to fully confirm that the book is wrong though Apr 11 '19 at 7:39