While experimenting with transformers' TFBertForSequenceClassification and BertTokenizer, I noticed that BertTokenizer:

transformer_bert_tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

tokenizes the text differently from the tokenizer that I use to construct for my BERT models in this way:

!wget --quiet https://raw.githubusercontent.com/tensorflow/models/master/official/nlp/bert/tokenization.py
import tokenization
FullTokenizer = tokenization.FullTokenizer

and then

BERT_MODEL_HUB = 'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2'
bert_layer = hub.KerasLayer(BERT_MODEL_HUB, trainable=True)
to_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
vocabulary_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()

tokenizer = FullTokenizer(vocabulary_file, to_lower_case)

as an example:

sequence = "Systolic arrays are cool. This 🐳 is cool too."
transformer_bert_tokenizer .tokenize(sequence)
# output: ['s', '##ys', '##to', '##lic', 'array', '##s', 'are', 'cool', '.', 'this', '[UNK]', 'is', 'cool', 'too', '.']
# output: ['sy','##sto','##lic','arrays','are','cool','.','this','[UNK]', 'is','cool','too','.']

Does anyone know why there is a difference? Aren't both tokenizers using the same vocabulary? which way is preferred?


Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary extraction process has also use lowercase text as input, and therefore gives as result a different vocabulary than the same vocabulary extraction process used with text in its original casing.

Note that, as the vocabularies are different, each model should be used with the tokenizer it was used to train it. Using a model with a different tokenizer may lead to bad results.

The choice of tokenizer, therefore, is tied to the choice of BERT model. The criteria to use one or the other (e.g. uncased vs. cased), depends on the case. For instance, for doing named entity recognition (NER) in English, it may be important to keep the original casing so that the model can more easily distinguish proper nouns.

  • $\begingroup$ Thank you very much for responding! I read your answer but still, I'm a bit confused that how these two can tokenize differently when both are set for the same task and the same model? The only difference that I see is the place I import them. but they are both designed for uncased bert tokenization. $\endgroup$ – mitra mirshafiee Nov 2 '20 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.