What's the intuition behind the hidden states of RNN/LSTM? Are they similar to the hidden states of HMM (Hidden Markov Model)?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
I personally don't think they are comparable to the hidden state of a Markov model. One key difference is that, in a HMM you can explain what a given state means to someone, where in a RNN/LSTM you cannot interpret a given state.
The closest thing that you can compare the hidden state of an RNN/LSTM is to think of it as the output of an intermediate layer of a fully-connected neural network but for time-series data.
And the larger the hidden state the more memory it can retain of the past.
$\begingroup$
$\endgroup$
Just to add, the hidden state can be described as the working memory of the recurrent network that carries information from immediately previous timesteps/events. This working memory overwrites itself at every step uncontrollably and is present at RNNs and LSTMs.
Given the latter, I appreciate the analogy with markovian framework - in a wider sense. Feel free to check my answer to a similar question for more information on the hidden and cell state architectures in sequence models.
$\begingroup$
$\endgroup$
I don't think you are too far off. Here's Geoff Hinton explaining his motivation for using the phrase 'hidden' in his work on neural nets in the late 80's:
The reason hidden units in neural nets are called hidden units is that Peter Brown told me about hidden Markov models. I decided "hidden" was a good name for those extra units, so that's where the name "hidden" comes from. (p. 379)
In the same article he also had this to say about the influence of statistical methods on the development of neural nets:
There's been a lot more use of the underlying algorithm in hidden Markov models; the EM algorithm is used much more in neural nets now. There's been a lot of transfer of ideas from statistics into neural networks. I think there's also transfer the other way. I think it's a very fruitful interaction, so that's something very good that's happened. (pp. 382-383)
Source: https://direct.mit.edu/books/book/4886/chapter/622920/Geoffrey-E-Hinton
The context of how "hidden" is used in LSTM specifically is slightly different. In the technical paper describing LSTM, Schmidhuber and Hochreiter repeatedly refer to the "conventional" or "standard" hidden units (see pg. 7, 8)- but notice, not hidden layers. This is used to show the difference between LSTM cells and "conventional" hidden units. Certainly by this point the usage of "hidden" in the neural nets world was rampant to refer to the hidden units, a la Hinton. Schmidhuber and Hochreiter also explicitly state:
In contrast to finite state automata or hidden Markov models LSTM does not require an a priori choice of a finite number of states. In principle it can deal with unlimited state numbers. (p. 23)
Source: https://www.bioinf.jku.at/publications/older/2604.pdf
To summarize, the intuition behind them is specific to LSTM: what is hidden is the nets learned state of what data/sequences matter and what doesn't. A HMM on the other hand assumes further (hidden/latent) Markov models nested inside it. The context is slightly different, although Hinton's comment on the flow between statistics and neural networks is apt.
