As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode the token as one single piece.
However, when I use pytorch
BertTokenizer it seems the encoder also separate token into pieces. Is this correct behaviour?
from pytorch_pretrained_bert import BertTokenizer, cached_path tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False) tokenizer.tokenize('<s> This is a sentence <|endoftext|>')
The results are:
['<', 's', '>', 'This', 'is', 'a', 'sentence', '<', '|', 'end', '##oft', '##ex', '##t', '|', '>']