NER with LSTM - How to recognize person names that are not part of the vocabulary?

Question

I am learning Named Entity Recognition and going through posts similar to this one:

Named-Entity Recognition (NER) using Keras Bidirectional LSTM

So the sentences are fed into the model as a sequence of integers - every int corresponding to the index in the vocabulary - from what I understand this is how the embedding layer works.

My question is - does that mean the model would not be able to recognize a person's name if it doesn't exist in the vocabulary?

For example, from the sentence:

"John Doe went for a walk"

given John Doe is in the vocabulary, it will be recognized as a person name but the sentence:

"Unknown Name went for a walk"

will not be properly tagged if Unknown Name is not in the vocab?

To me, this would be a little strange as Unknown Name is in the same context as John Doe so I was hoping to be able to somehow tag it properly.

I'm obviously lacking knowledge here so I'd be very grateful for any suggestions and reference materials.

Answer 1

With word-level vocabularies, unknown words are normally encoded with a special token <unk>. If in the training data there were enough examples of person names, then the bidirectional LSTM may have learned to identify the person name from the context, not just the word itself. In that case, and provided that in the specific input sentence there is enough context, the model may be able to identify it. If the model were a normal LSTM instead of a bidirectional one, it would be more difficult for the model to identify it, as it would only be able to use the context to the left of the unknown word.

If the training data did not contain such examples, the it won't be able to identify the person from the context.

One option to improve the handing of this problem would be to force this kind of examples in the training data, by replacing person names with unknown words with certain probability.

Another option would be to use a subword-level vocabulary, like a byte-pair encoding (BPE) one, eliminating altogether the out-of-vocabulary (OOV) word problem.