I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further downstream task.

Bert's last layer looks like this :

enter image description here

Where we take the [CLS] token of each sentence :

enter image description here

Image source

I went through many discussion on this huggingface issue, datascience forum question, github issue Most of the data scientist gives this explanation :

BERT is bidirectional, the [CLS] is encoded including all representative information of all tokens through the multi-layer encoding procedure. The representation of [CLS] is individual in different sentences.

My question is, Why the author ignored the other information ( each token's vector ) and taking the average, max_pool or other methods to make use of all information rather than using [CLS] token for classification?

How does this [CLS] token help compare to the average of all token vectors?


3 Answers 3


It's because you need to fine-tune BERT for your specific task anyway. You can train it to classify based on either cls token, or mean of token outputs, or whatever.

In essence, CLS token of the last layer has connections with all of the other tokens on the previous layer. So, does it make sense to average manually?

  • $\begingroup$ Thanks for your answer. What do you mean by anyway? I can use CLS token or Mean of token outputs as embedding vector even in the pre-trained BERT model. Am i wrong? $\endgroup$
    – canP
    Commented Jul 12, 2022 at 21:12
  • $\begingroup$ @canP you're right that any method works as long as you're finetuning your model. However, with a pre-trained model without fine-tuning, there is no proper answer. It depends on how the model was trained. $\endgroup$
    – roman
    Commented Jul 13, 2022 at 16:27

The CLS token helps with the NSP task on which BERT is trained (apart from MLM). The authors found it convenient to create a new hidden state at the start of a sentence, rather than taking the sentence average or other types of pooling. However this does not mean that the BERT authors recommend using the CLS token as a sentence embedding. It 'could' be used for classification or other tasks, but there could be other poolers you may want to write yourself using the embeddings of the individual words. In my experience max pooling works for sentiment analysis. For other NLU types of tasks mean pooling works better. In other cases an extra attention head and a fine-tuning can lead to excellent results. Each problem domain could have a specific solution.


Not sure what it meant -> rather than using [CLS] token for classification?

The Authoer did use [CLS] for classification tasks.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

The first token of every sequence is always a special classification token ([CLS]). The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks.

For instance


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.