I am working on multilingual (English, Arabic, Chinese) NER and I met a problem: how to tokenize data?

My train data provides sentence and list of spans for each named entity.


[('The', 'DT'),
 ('company', 'NN'),
 ('said', 'VBD'),
 ('it', 'PRP'),
 ('believes', 'VBZ'),
 ('the', 'DT'),
 ('suit', 'NN'),
 ('is', 'VBZ'),
 ('without', 'IN'),
 ('merit', 'NN'),
 ('.', '.')]

[('你', 'PN'),
 ('有', '.'),
 ('没', 'AD'),
 ('有', '.'),
 ('用', 'VV'),
 ('过', '.'),
 ('其它', 'DT'),
 ('药品', 'NN'),
 ('?', 'PU')]

What the best way to tokenize input data? There are main different alternatives I consider: word level, wordpiece level, BPE.

BPE does not work with Chinese and Arabic because of unicode thing. I doubt about word level because I am not sure what is a word in Chinese.

What can you recommend me?


1 Answer 1


First, some clarification: BPE does work with Chinese and Arabic.

The only problem with Chinese is that there are no blanks between words, and therefore there is no explicit word boundary. In order to address that problem, normally you would segment words before applying BPE. For that, the typical approach is to use Jieba or any of its multiple ports to other programming languages. Other languages without blanks, like Japanese, may have their own tools to perform word segmentation.

Now, the answer:

Subword vocabularies are the norm nowadays. The norm also consists of finetuning one of the many pre-trained neural models available (BERT, XLNet, RoBERTa, etc), and the tokenization is imposed by the model you choose.

BPE is, in general, a popular choice for tokenization, no matter the language. Lately, the unigram tokenization is becoming popular also.

I suggest you take a look at the recent tner library, which builds on top of Huggingface Transformers to make NER very easy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.