Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

Question

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further downstream task.

Bert's last layer looks like this :

Where we take the [CLS] token of each sentence :

Image source

I went through many discussion on this huggingface issue, datascience forum question, github issue Most of the data scientist gives this explanation :

BERT is bidirectional, the [CLS] is encoded including all representative information of all tokens through the multi-layer encoding procedure. The representation of [CLS] is individual in different sentences.

My question is, Why the author ignored the other information ( each token's vector ) and taking the average, max_pool or other methods to make use of all information rather than using [CLS] token for classification?

How does this [CLS] token help compare to the average of all token vectors?

Answer 1

It's because you need to fine-tune BERT for your specific task anyway. You can train it to classify based on either cls token, or mean of token outputs, or whatever.

In essence, CLS token of the last layer has connections with all of the other tokens on the previous layer. So, does it make sense to average manually?

Answer 2

The CLS token helps with the NSP task on which BERT is trained (apart from MLM). The authors found it convenient to create a new hidden state at the start of a sentence, rather than taking the sentence average or other types of pooling. However this does not mean that the BERT authors recommend using the CLS token as a sentence embedding. It 'could' be used for classification or other tasks, but there could be other poolers you may want to write yourself using the embeddings of the individual words. In my experience max pooling works for sentiment analysis. For other NLU types of tasks mean pooling works better. In other cases an extra attention head and a fine-tuning can lead to excellent results. Each problem domain could have a specific solution.

Answer 3

Not sure what it meant -> rather than using [CLS] token for classification?

The Authoer did use [CLS] for classification tasks.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

The first token of every sequence is always a special classification token ([CLS]). The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks.

For instance