3
$\begingroup$

As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode the token as one single piece.

However, when I use pytorch BertTokenizer it seems the encoder also separate token into pieces. Is this correct behaviour?

from pytorch_pretrained_bert import BertTokenizer, cached_path
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False) 
tokenizer.tokenize('<s> This is a sentence <|endoftext|>')

The results are:

['<',
 's',
 '>',
 'This',
 'is',
 'a',
 'sentence',
 '<',
 '|',
 'end',
 '##oft',
 '##ex',
 '##t',
 '|',
 '>']
Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained representations if you keep them. You should remove these special tokens from the input text.

In the case of GPT-2, OpenAI trained it only with <|endoftext|>, but it has to be added after the tokenization. Some people mistakenly add it before tokenization, leading to problems. <|startoftext|> is specific to the library gpt-2-simple.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Or we can also extend the vocab to add the new token's depending on the task? $\endgroup$
    – Aditya
    Commented Jan 13, 2020 at 16:36
  • $\begingroup$ Extending the vocabulary of an already trained model is normally not a good idea (apart from being technically challenging due to the differences in tensor sizes). Also, your examples of special tokens don't add anything new, so I see no point in trying hard to keep them. $\endgroup$
    – noe
    Commented Jan 13, 2020 at 22:26
  • $\begingroup$ As I read from some technical blogs, typically they will add these tokens as a sentence separator. So I am confused now $\endgroup$
    – Kevin Ling
    Commented Jan 14, 2020 at 3:46
  • $\begingroup$ Or if another question, other than the normal punctuation, is there any way I can add in some special sentence separator? $\endgroup$
    – Kevin Ling
    Commented Jan 14, 2020 at 3:47
  • 1
    $\begingroup$ @Aditya if you are finetuning BERT on data that use those special tokens for such specific purposes, it may work. If you are taking BERT's weights as is and expect that using those tokens in different ways from what BERT was trained on, I would not expect good results. $\endgroup$
    – noe
    Commented Jan 15, 2020 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.