I'm very terrible at NLP and I have searched for these questions but didn't find any answer, my question is, in RNNs, there are hidden states to remember information for processing the next state, and in Transformers, there are also hidden states for each attention layer. Are these different or the same ? Additionally, I often read that RNNs or LSTMs have a dimension of $256$, while Transformers have a dimension of $768$. What are the meaning of these numbers, what are they used for ? Thanks


1 Answer 1


The hidden states in RNNs and Transformers share their purpose, but they work in different ways.

  • In RNNs, the hidden state is a way for the network to maintain a memory of previous time steps. The hidden state at a given time step is calculated based on the input at the current time step and the hidden state from the previous time step.
  • Transformers use a self-attention mechanism to capture dependencies between different positions in the input sequence. They process the entire input sequence in parallel, considering all positions simultaneously. "Hidden states" in transformers is normally used to refer to the output of each attention layer.

Both RNNs and Transformers work on numeric vectors. Text is discrete, so we normally use embedding layers to represent discrete text tokens (e.g. words) as numeric vectors. The size of these vectors is defined a prior, when designing the network. The numbers you mentioned are typical values of the embedding dimensionality (e.g. BERT's embedding dimensionality is 768), but any value can be used. Of course, the chosen value has an effect on the capacity of the network and it's ability to learn.

  • $\begingroup$ Thank you, I want to ask what does the output of each attention layer of transformer contain ? Also dimensionality is something like 1 word or token has $768$ features vectors, apply for every words or tokens in the vocabulary ? $\endgroup$
    – user159173
    Jan 31 at 21:14
  • $\begingroup$ The output of each attention layer contains one vector per each of the input tokens; for instance, in BERT, if the input has 12 tokens, the output of each attention layer has 12 vectors of 768 components each. The vectors are computed for the specific inputs, unlike word embeddings that always have the same vector for a specific word regardless from the rest of the sentence (i.e. context) where the word appears. $\endgroup$
    – noe
    Feb 1 at 6:51
  • $\begingroup$ Thank you, so word embeddings in transformers are different from RNNs in terms of contextualization ? I thought RNNs or LSTMs process 1 input at a time, so they remember the position of a word in a sentence $\endgroup$
    – user159173
    Feb 1 at 20:49
  • $\begingroup$ When I mentioned "word embeddings" I wasn't referring to RNNs/LSTMs, but to non-contextual word embeddings like word2vec. $\endgroup$
    – noe
    Feb 1 at 20:54
  • $\begingroup$ Thanks, when you said in transformers that "they process the entire input sequence in parallel, considering all positions simultaneously", there is a question about this, I don't know the answer. Could you please clarify this for me ? Thanks $\endgroup$
    – user159173
    Feb 1 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.