Recently, I read about one of the state-of-the-art method called Attention models. This method use a Encoder-Decoder model. It can find a better encoding for each word in a sentence. But how can I encode a full sentence?

For example, I have a sentence "I love reading".

After embedding, this sentence will be converted to list of three vectors. (or matrix with dimension number of words times embedding dimension).

After several layers of attention mechanism, I will still have the same matrix.

How can I convert this matrix to a single vector that contains an encoded representation of the full sentence?


2 Answers 2


A standard way of obtaining a sentence representation with attention models is using BERT or any other of its derivations, like RoBERTa. In these models, the sentence tokens passed as input to the model are prefixed with a special token [CLS]. The output of the model at that first position is the sentence representation.

To use these models, you may use sentence-transformers library, e.g.:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('paraphrase-distilroberta-base-v1')

sentences = ['This framework generates embeddings for each input sentence',
    'Sentences are passed as a list of string.', 
    'The quick brown fox jumps over the lazy dog.']

sentence_embeddings = model.encode(sentences)

for sentence, embedding in zip(sentences, sentence_embeddings):
    print("Sentence:", sentence)
    print("Embedding:", embedding)
  • $\begingroup$ Is is possible to get the same from reformer model? $\endgroup$ Commented Jun 1, 2021 at 23:22
  • $\begingroup$ It is technically possible, but you need to find a reformer pretrained on a masked LM task. I am not aware of the existence of such a publicly available pretrained model. $\endgroup$
    – noe
    Commented Jun 2, 2021 at 9:57

How can I convert this matrix to a single vector that contains an encoded representation of the full sentence?

It's achieved by applying a softmax function to the attention scores and using these probabilities to derive a weighted sum of the encoder hidden states.

More specifically: In Seq2Seq, for example, with $N$ words let $h_1,...,h_N \in R^h$ be the encoder hidden states, $s_t \in R^h$ the decoder hidden states at timestep $t$, then the attention scores are $$e^t = [s_t^Th_1,...,s_t^Th_N] \in R^N.$$ Applying a softmax function gives the attention distribution $$\alpha^t = softmax(e^t) \in R^N.$$ And, finally, using these as weights in a weighted sum results in the attention output $$a_t = \sum_{i=1}^N \alpha_i^t h_i \in R^h.$$

By doing so, you reduce the sequence of $N$ words to a single vector of dimension $h$. This is also well explained in Stanford's CS224N: Natural Language Processing with Deep Learning - Lecture 8 (around 1h 2mins).

  • $\begingroup$ This list of vectors already is a result of QKV-attention. The question was about how to get a single encoding for a full sentence, not encoding for each word. $\endgroup$ Commented Jun 1, 2021 at 21:06
  • $\begingroup$ @KenenbekArzymatov Not sure what you mean. $\alpha_t$ encodes the whole input sentence, not each word. Moreover, what I described is the attention mechanism. The input vectors $s_t$ and $h_n$ are the inputs to and not the result of attention (except for this being repeated over time steps, of course). $\endgroup$
    – Jonathan
    Commented Jun 2, 2021 at 7:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.