NER with Unsupervised Learning?

Question

If we treated NER as a classification/prediction problem, how would we handle name entities that weren't in training corpus?

For example, "James was born in England." James was labeled as a PERSON and England as a LOCATION. But we type another completely strange sentence like "Fyonair is from Fuabalada land." We as human can understand Fyonair is a person (or maybe princess from a fairy tale) and Fuabalada is the land where she comes from.

How would our model identify it if it wasn't included in billions of corpus and tokens? Can unsupervised learning achieve this task?

Answer 1

If we treated NER as a classification/prediction problem, how would we handle name entities that weren't in training corpus?

The goal of a NER Tagger is to learn patterns in language that can be used to classify words (or more generally, tokens), given a pre-specified set of classes. These classes are defined before training and remain fixed. Classes such as: PERSON, DATETIME, ORGANIZATION, ... you name it.

A good NER Tagger will learn the structure of a language and recognize that "Fyonair is from Fuabalada land." follows some linguistic rules and regularities, and that from these regularities (learned autonomously during training) the classifier can attribute Fyonair class PERSON and to Fuabalada the class LOCATION.

---

How would our model identify it if it wasn't included in billions of corpus and tokens?

In fact, Deep Learning models tend to work better than others with very large datasets (the so called "big data"). On small datasets they are not extremely useful.

---

Can unsupervised learning achieve this task?

NER tagging is a supervised task. You need a training set of labeled examples to train a model for that. However, there is some unsupervised work one can do to slightly improve the performance of models. There is this useful paragraph that I took from Geron's book:

Suppose you want to tackle a complex task for which you don't have much labeled training data [...] If you can gather plenty of unlabeled training data, you can try to use it to train an unsupervised model, such as an autoencoder or a generative adversarial network [...] Then you can reuse the lower layers of the autoencoder or the lower layers of the GAN's discriminator, add the output layer for your task on top, and fine tune the final network using supervised learning (i.e. the label training examples).

It is this technique that Geoffrey Hinton and his team used in 2006 and which led to the revival of neural network and the success of Deep Learning.

[ p. 349, 2nd edition. ]

(Best book on Machine Learning ever, IMHO.)

This unsupervised pretraining is the only way to use unsupervised models for NER that I can think of.

Good luck with your task!

Answer 2

First, when we as humans interpret "Fyonair is from Fuabalada land" we use our knowledge of "is from" and "X land" to infer that Fyonair is probably a person and Fuabalada land probably a location. Therefore our process is not (at least not completely) unsupervised: we have seen this sentence structure before ("training") and we use our "model" to "predict" the two entities.

A good (supervised) NER is trained to recognize this kind of patterns as well. The example that you mention might be a bit too hard, but with something like "Dr Fyonair is the new CEO of Fubalada" a decent NER should be able to recognize that "Dr Fyonair" is a person (thanks to the "Dr") and that "Fubalada" is a company (thanks to the "CEO of") even though it has never seen these particular proper names.

So a standard (supervised) NER is supposed to recognize entities it has never seen before, provided it has clues in the sentence about them. Technically it's not really a state of the art NER if it recognizes only the entities that it has seen during training, it's just a string matching program. It's true that NERs make a lot of errors with entities which haven't been seen previously, but that's simply because they are harder to catch in general.