I am working on multilingual (English, Arabic, Chinese) NER and I met a problem: how to tokenize data?
My train data provides sentence and list of spans for each named entity.
e.g.
[('The', 'DT'),
('company', 'NN'),
('said', 'VBD'),
('it', 'PRP'),
('believes', 'VBZ'),
('the', 'DT'),
('suit', 'NN'),
('is', 'VBZ'),
('without', 'IN'),
('merit', 'NN'),
('.', '.')]
[('你', 'PN'),
('有', '.'),
('没', 'AD'),
('有', '.'),
('用', 'VV'),
('过', '.'),
('其它', 'DT'),
('药品', 'NN'),
('?', 'PU')]
What the best way to tokenize input data? There are main different alternatives I consider: word level, wordpiece level, BPE.
BPE does not work with Chinese and Arabic because of unicode thing. I doubt about word level because I am not sure what is a word in Chinese.
What can you recommend me?