How does BERT produce CLS token? Internally does it do max-pooling or avarage pooling?

Question

I ran experiment to compare max-pooled word tokens vs CLS token for sentence classification and CLS clearly wins. Trying to understand how BERT generates CLS token embedding if its better than max or avg pooling.

Answer 1

The output at the first position (which is the position the special token [CLS] is at the input sequence and is what you call the "CLS token") is neither computed with max-pooling or average pooling, but it is computed with self-attention, like the other output positions. The difference with the other output positions is that the first position is trained with the next sentence prediction (NSP) task. This means that the representation learned there is meant to predict whether the second part of the input (the subsequence after the [SEP] special token) was following the first part of the input in the original document.

You can check the details at section 3.1 of the original BERT paper, within the "Task #2: Next Sentence Prediction (NSP)" subtitle. The following figure from the paper illustrates how the output at the first position is used for the NSP task: