What is the use of [SEP] in paper BERT?

Question

I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun.

However, I have a question.

If I have 2 sentences, which are s1 and s2, and our fine-tuning task is the same.

In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP].

In another, I make the input look like [CLS] + s1 + s2 + [SEP].

When I input them to BERT respectively, what is the difference between them?

• Will the s1 in second one integrate more information from s2 than the s1 in first one does?

• Will the token embeddings change a lot between the 2 methods?

Thanks for any help!

Answer 1

Neural networks are black boxes and any answer would be pure speculation, especially not knowing what fine tuning task we are speaking about.

Only by running an experiment would you get the actual answer, and not even then, given the loose definition of the formulations "integrate more information" and "change a lot" in your questions.

Answer 2

First let's understand why the format is like this.

BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not.
The [SEP] in the middle is here to help the model understand which token belong to which sentence.

At finetuning time, if you use a different format than pretraining format, you might confuse the model : he never saw 2 sentences formatted as [CLS] Sen A Sen B [SEP]. The model doesn't know there is 2 sentences, and will consider it as a single sentence.

If you finetune on enough data, BERT can learn the new format. This can be helpful if you need to change the input format.

But in your case, you don't need to do this. Changing the format for the sake of changing the format is just going to confuse your model, he will have to learn more thing, and there will be inconsistencies between pretraining and finetuning.

---

Will the s1 in second one integrate more information from s2 than the s1 in first one does?

No. Inserting a SEP token or not will not change the amount of information exchange between the tokens of the 2 sentences. In both case the model will compute attention based on the 2 sentences. Each sentence can see the other sentence's tokens, no matter of the SEP.
The only thing you will do by removing the SEP token is confuse your model.

Will the token embeddings change a lot between the 2 methods?

We don't know. It will definitely change, but how much ? We cannot answer. My guess is that token representations will not change a lot (because tokens are the same), but CLS representation will change a lot (instead of representing the links between 2 sentences, it will represent something else).