First let's understand why the format is like this.
BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]
. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not.
The [SEP]
in the middle is here to help the model understand which token belong to which sentence.
At finetuning time, if you use a different format than pretraining format, you might confuse the model : he never saw 2 sentences formatted as [CLS] Sen A Sen B [SEP]
. The model doesn't know there is 2 sentences, and will consider it as a single sentence.
If you finetune on enough data, BERT can learn the new format. This can be helpful if you need to change the input format.
But in your case, you don't need to do this. Changing the format for the sake of changing the format is just going to confuse your model, he will have to learn more thing, and there will be inconsistencies between pretraining and finetuning.
Will the s1 in second one integrate more information from s2 than the s1 in first one does?
No. Inserting a SEP
token or not will not change the amount of information exchange between the tokens of the 2 sentences. In both case the model will compute attention based on the 2 sentences. Each sentence can see the other sentence's tokens, no matter of the SEP
.
The only thing you will do by removing the SEP
token is confuse your model.
Will the token embeddings change a lot between the 2 methods?
We don't know. It will definitely change, but how much ? We cannot answer. My guess is that token representations will not change a lot (because tokens are the same), but CLS
representation will change a lot (instead of representing the links between 2 sentences, it will represent something else).