After reading around on the web I came to understand that the output representation of the special token [CLS] captures the representation of a sentence (am I correct?).

My primary question is what information does the output embedding of [SEP] token (T_SEP) captures?

My other doubt is if I input a bunch of sentences into BERT separated by [SEP] does the output embedding of [CLS] contain information about all the sentences?


1 Answer 1


You are right, CLS token tries to capture the sentence representation, because during pretraining this token is used to decide if 2 sentences are contiguous or not.

But the author mentioned that CLS token was not supposed to be a sentence representation, and should be used carefully.

The SEP token is a token used simply to separate sentence, in order to make is easier for BERT to know that the input is made of several sentences. Since the SEP token is not used at pretraining time, the SEP token does not represent anything.

About your last question : We don't know.

In pretraining, the model was not train with several sentence, so the model will not know how to handle several sentence.

But if you finetune it, the model may be able to learn new representation and can learn so that the CLS token contain information about all the sentences.

For example, this code finetuned BERT with a new pattern :

[CLS] Sen 1 [SEP] [CLS] Sen 2 [SEP] [CLS] Sen 2 [SEP] ...

And the CLS token is used to represent each sentence. Because the model is finetuned, BERT is learning to represent each sentence in the corresponding CLS token.

  • 1
    $\begingroup$ Yeah I was working with BERTSUM for extractive text summarization and in the paper the [CLS] token embedding were used to predict if that sentence will remain in the final summary. The inference I got from your answer was that the [CLS] token has a meaning only because of the way it was used during training. Thanks it will be helpful when I am studying other models. $\endgroup$ Commented May 10, 2020 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.