6
$\begingroup$

I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun.

However, I have a question.

If I have 2 sentences, which are s1 and s2, and our fine-tuning task is the same.

In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP].

In another, I make the input look like [CLS] + s1 + s2 + [SEP].

When I input them to BERT respectively, what is the difference between them?

  • Will the s1 in second one integrate more information from s2 than the s1 in first one does?

  • Will the token embeddings change a lot between the 2 methods?

Thanks for any help!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
10
$\begingroup$

First let's understand why the format is like this.

BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not.
The [SEP] in the middle is here to help the model understand which token belong to which sentence.

At finetuning time, if you use a different format than pretraining format, you might confuse the model : he never saw 2 sentences formatted as [CLS] Sen A Sen B [SEP]. The model doesn't know there is 2 sentences, and will consider it as a single sentence.

If you finetune on enough data, BERT can learn the new format. This can be helpful if you need to change the input format.

But in your case, you don't need to do this. Changing the format for the sake of changing the format is just going to confuse your model, he will have to learn more thing, and there will be inconsistencies between pretraining and finetuning.

Will the s1 in second one integrate more information from s2 than the s1 in first one does?

No. Inserting a SEP token or not will not change the amount of information exchange between the tokens of the 2 sentences. In both case the model will compute attention based on the 2 sentences. Each sentence can see the other sentence's tokens, no matter of the SEP.
The only thing you will do by removing the SEP token is confuse your model.

Will the token embeddings change a lot between the 2 methods?

We don't know. It will definitely change, but how much ? We cannot answer. My guess is that token representations will not change a lot (because tokens are the same), but CLS representation will change a lot (instead of representing the links between 2 sentences, it will represent something else).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for this answer. I have a question in my mind though, you said "we will confuse the model" Why would it be confused if we are already separating two sentences with "." ? $\endgroup$
    – canP
    Jul 14 at 4:55
  • 1
    $\begingroup$ @canP If all your sentences are separated with ".", then you might be right, maybe adding or removing SEP token wouldn't change that much. However, this might not be true : some sentences are separated by other punctuation (?, !, etc...). In some case, the text after tokenization might even not have punctuation at all ! In these case SEP token might be necessary. $\endgroup$
    – Astariul
    Jul 15 at 1:24
2
$\begingroup$

Neural networks are black boxes and any answer would be pure speculation, especially not knowing what fine tuning task we are speaking about.

Only by running an experiment would you get the actual answer, and not even then, given the loose definition of the formulations "integrate more information" and "change a lot" in your questions.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.