71
$\begingroup$

I am reading this article on how to use BERT by Jay Alammar and I understand things up until:

For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we select that slice of the cube and discard everything else.

I have read this topic, but still have some questions:

Isn't the [CLS] token at the very beginning of each sentence? Why is that "we are only interested in BERT's output for the [CLS] token"? Can anyone help me get my head around this? Thanks!

$\endgroup$

4 Answers 4

34
$\begingroup$

CLS stands for classification and its there to represent sentence-level classification.

In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog where this is also covered in detail.

$\endgroup$
3
  • $\begingroup$ The article you shared is quite helpful. Thanks! $\endgroup$ Jan 9, 2020 at 18:14
  • 5
    $\begingroup$ I think an answer should be standalone, and would appreciate the gist of the link, which will inevitably be broken in the future. $\endgroup$
    – Gulzar
    Aug 31, 2022 at 11:22
  • $\begingroup$ aaaaaand the images of the link are now broken 😅 $\endgroup$
    – Tanguy
    Jun 30, 2023 at 8:12
25
$\begingroup$

[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire sentence, they introduce a new tag.

They can’t take any other word from the input sequence, because the output of that is the word representation. So they add a tag that has no other purpose than being a sentence-level representation for classification.

$\endgroup$
19
$\begingroup$

In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks:

  1. Masked language modeling: some random words are masked with [MASK] token, the model learns to predict those words during training. For that task we need the [MASK] token.
  2. Next sentence prediction: given 2 sentences, the model learns to predict if the 2nd sentence is the real sentence, which follows the 1st sentence. For this task, we need another token, output of which will tell us how likely the current sentence is the next sentence of the 1st sentence. And here comes the [CLS]. You can think about the output of [CLS] as a probability.

Now you may ask the question: can we instead of using [CLS]'s output just outputting a number (as probability)? Yes, we can do that if the task of predicting next sentence is a separate task. However, BERT has been trained on both tasks simultaneously. Organizing inputs and outputs in such a format (with both [MASK] and [CLS]) will help BERT to learn both tasks at the same time and boost its performance.

When it comes to classification task (e.g. sentiment classification), as mentioned in other answers, the output of [CLS] can be helpful because it contains BERT's understanding at the sentence-level.

$\endgroup$
1
  • $\begingroup$ As my understanding CLS token is representation of whole text (sentence1 and sentence2), which means that model got trained such a way that CLS token is having probablity of "if second sentence is next sentence of 1st sentence", so how are people can generate sentence embeddings from CLS tokens? How does CLS is having the meaning of the sentence in a single sentence classification task without even being trained for it?? $\endgroup$
    – canP
    Jul 14, 2022 at 4:02
15
$\begingroup$

Here're my understandings:

(1)[CLS] appears at the very beginning of each sentence, it has a fixed embedding and a fix positional embedding, thus this token contains no information itself. (2)However, the output of [CLS] is inferred by all other words in this sentence, so [CLS] contains all information in other words.

This makes [CLS] a good representation for sentence-level classification.

of course, you may use an average vector, it makes sense, too.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.