I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun.

However, I have a question.

If I have 2 sentences, which are s1 and s2, and our fine-tuning task is the same.

In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP].

In another, I make the input look like [CLS] + s1 + s2 + [SEP].

When I input them to BERT respectively, what is the difference between them?

  • Will the s1 in second one integrate more information from s2 than the s1 in first one does?

  • Will the token embeddings change a lot between the 2 methods?

Thanks for any help!


2 Answers 2


First let's understand why the format is like this.

BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not.
The [SEP] in the middle is here to help the model understand which token belong to which sentence.

At finetuning time, if you use a different format than pretraining format, you might confuse the model : he never saw 2 sentences formatted as [CLS] Sen A Sen B [SEP]. The model doesn't know there is 2 sentences, and will consider it as a single sentence.

If you finetune on enough data, BERT can learn the new format. This can be helpful if you need to change the input format.

But in your case, you don't need to do this. Changing the format for the sake of changing the format is just going to confuse your model, he will have to learn more thing, and there will be inconsistencies between pretraining and finetuning.

Will the s1 in second one integrate more information from s2 than the s1 in first one does?

No. Inserting a SEP token or not will not change the amount of information exchange between the tokens of the 2 sentences. In both case the model will compute attention based on the 2 sentences. Each sentence can see the other sentence's tokens, no matter of the SEP.
The only thing you will do by removing the SEP token is confuse your model.

Will the token embeddings change a lot between the 2 methods?

We don't know. It will definitely change, but how much ? We cannot answer. My guess is that token representations will not change a lot (because tokens are the same), but CLS representation will change a lot (instead of representing the links between 2 sentences, it will represent something else).

  • $\begingroup$ Thanks for this answer. I have a question in my mind though, you said "we will confuse the model" Why would it be confused if we are already separating two sentences with "." ? $\endgroup$
    – canP
    Commented Jul 14, 2022 at 4:55
  • 1
    $\begingroup$ @canP If all your sentences are separated with ".", then you might be right, maybe adding or removing SEP token wouldn't change that much. However, this might not be true : some sentences are separated by other punctuation (?, !, etc...). In some case, the text after tokenization might even not have punctuation at all ! In these case SEP token might be necessary. $\endgroup$
    – Astariul
    Commented Jul 15, 2022 at 1:24
  • 1
    $\begingroup$ @Astariul As I understand, "[CLS] s1 [SEP] s2 [SEP]" is the default input structure of BERT. Is it right? By any chance, if the s3 exits, and the structure "[CLS] s1 [SEP] s2 [SEP] s3 [SEP]", is it works? (I dont have specific task with that input, just for knowledge) Or, that abnormal structure will also count as "New format" and the model will learn it if it has enough data, just not recommended. $\endgroup$ Commented Mar 7, 2023 at 8:47
  • 1
    $\begingroup$ @TrungTínTrần It's difficult to say without details about the task... My guess is that if s3 is used similarly as s2, this might work. But anyway, the best is to finetune the model on your specific dataset and task, I'm convinced you'll get better results. $\endgroup$
    – Astariul
    Commented Mar 8, 2023 at 13:19

Neural networks are black boxes and any answer would be pure speculation, especially not knowing what fine tuning task we are speaking about.

Only by running an experiment would you get the actual answer, and not even then, given the loose definition of the formulations "integrate more information" and "change a lot" in your questions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.