# Questions about hidden states of bidirectional LSTMs

In bidirectional LSTM, each word corresponds to two hidden states, one for each direction. Thus, we concatenate these two hidden states to represent the semantic meaning of a word. Additionally, the last hidden states of the bidirectional LSTM are concatenated to be the sentence vector

Could someone explain what are these two hidden states representing each word, and also clarify what are the last hidden states when representing the sentence?

Let's take a look at the following diagram (source):

A bidirectional LSTM consists of two independent LSTMs $$A$$ (green) and $$A'$$ (darker green). The input sequence $$X_0...X_i$$ (blue) is passed to each of the LSTMs, but for $$A$$ we process it in normal order $$X_0...X_i$$ and for $$A'$$ we process it in reverse order $$X_i...X_0$$.

The outputs of each LSTM at a specific position are concatenated together, obtaining a sequence of hidden states $$Y_0,...Y_i$$, one per token in the input sequence.

The hidden state after the final token of each LSTM $$S_i$$ (pink) and $$S'_i$$ (darker pink) can also be concatenated (not in the diagram), obtaining a vector representing the whole sequence.

The initial hidden states $$S_0$$ and $$S'_0$$ are usually initialized to zeroes.

• So these word and sentence vectors are contextual, right ?
– user159173
Commented Feb 10 at 9:14
• Yes. Actually, bidirectional LSTMs where used for contextual representations for ELMo before BERT.
– noe
Commented Feb 10 at 9:22
• (I'm not talking about ELMo) I thought the process was like this: you feed the word and sentence embeddings obtained from models like word2vec into a bidirectional LSTM. However, can a bidirectional LSTM alone also obtain these embeddings?
– user159173
Commented Feb 10 at 9:32
• The final tokens of each LSTM $S_i$ and $S'_i$ can also be concatenated, do you mean to concatenate $S_i$ and $S'_i$ into a single vector? Additionally, when you mention these tokens are not in the diagram, do you mean the tokens represented by darker pink and pink circles in the diagram ?
– user159173
Commented Feb 10 at 10:50
• The hidden state ($S_t$ in the figure) passed from one timestep to the following has 2 components: $h_t$ and $c_t$. The output at a timestamp ($Y_t$ in the figure) is $h_t$. I suggest you check this blog post for an introduction to LSTMs.
– noe
Commented Feb 17 at 22:31