Why BERT tokenizers function differently?

Question

While experimenting with transformers' TFBertForSequenceClassification and BertTokenizer, I noticed that BertTokenizer:

transformer_bert_tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

tokenizes the text differently from the tokenizer that I use to construct for my BERT models in this way:

!wget --quiet https://raw.githubusercontent.com/tensorflow/models/master/official/nlp/bert/tokenization.py
import tokenization
FullTokenizer = tokenization.FullTokenizer

and then

BERT_MODEL_HUB = 'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2'
bert_layer = hub.KerasLayer(BERT_MODEL_HUB, trainable=True)
to_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
vocabulary_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()

tokenizer = FullTokenizer(vocabulary_file, to_lower_case)

as an example:

sequence = "Systolic arrays are cool. This 🐳 is cool too."
transformer_bert_tokenizer .tokenize(sequence)
# output: ['s', '##ys', '##to', '##lic', 'array', '##s', 'are', 'cool', '.', 'this', '[UNK]', 'is', 'cool', 'too', '.']
tokenizer2.tokenize(sequence)
# output: ['sy','##sto','##lic','arrays','are','cool','.','this','[UNK]', 'is','cool','too','.']

Does anyone know why there is a difference? Aren't both tokenizers using the same vocabulary? which way is preferred?

Answer 1

Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary extraction process has also use lowercase text as input, and therefore gives as result a different vocabulary than the same vocabulary extraction process used with text in its original casing.

Note that, as the vocabularies are different, each model should be used with the tokenizer it was used to train it. Using a model with a different tokenizer may lead to bad results.

The choice of tokenizer, therefore, is tied to the choice of BERT model. The criteria to use one or the other (e.g. uncased vs. cased), depends on the case. For instance, for doing named entity recognition (NER) in English, it may be important to keep the original casing so that the model can more easily distinguish proper nouns.