# What should be the labels for subword tokens in BERT for NER task?

For any NER task, we need a sequence of words and their corresponding labels. To extract features for these words from BERT, they need to be tokenized into subwords.

For example, the word 'infrequent' (with label B-count) will be tokenized into ['in', '##fr', '##e', '##quent']. How will its label be represented?

According to the BERT paper, "We use the representation of the first sub-token as the input to the token-level classifier over the NER label set".

So I assume, for the subwords ['in', '##fr', '##e', '##quent'] , the label for the first subword will either be this ['B-count', 'B-count', 'B-count', 'B-count'] where we propagate the word label to all the subwords. Or should it be ['B-count', 'X', 'X', 'X'] where we leave the original label on the first token of the word, then use the label “X” for subwords of that word.

Any help will be appreciated.

• as far as I've understood, you should map only the root of the word that gets "tokenized" in subwords. I'm facing similar issues with a WSD task. After BERT, you should make a custom layer that removes the subwords... how to do so is still a mystery to me – Gianmarco F. Mar 15 '20 at 18:53
• Yes, we need to combine the subword token embeddings using some method (averaging over them?) and get rid of the dummy (X) subword token labels. I need to figure that out too yet. – adjective_noun Mar 17 '20 at 13:38

This is also the reason why we don't use method 1 is that otherwise, we would be introducing more labels of the type [B-count] and affecting the support number for such a class (which would make a test set no longer comparable with other models that do not increase the number of labels for such class).