# Word2Vec for Named Entity Recognition

I'm looking to use google's word2vec implementation to build a named entity recognition system. I've heard that recursive neural nets with back propagation through structure are well suited for named entity recognition tasks, but I've been unable to find a decent implementation or a decent tutorial for that type of model. Because I'm working with an atypical corpus, standard NER tools in NLTK and similar have performed very poorly, and it looks like I'll have to train my own system.

In short, what resources are available for this kind of problem? Is there a standard recursive neural net implementation available?

• Have you tried training the Stanford NER on your corpus? There's a tutorial here. – Emre Jun 19 '14 at 20:58
• I have not -- should give that a go to see how it fares. – Madison May Jun 19 '14 at 21:30
• I'd like to use word2vec features or similar, though, since I have access to a relatively small labeled dataset and need to make the most of unlabeled data I have on hand. – Madison May Jun 19 '14 at 21:37

Instead of "recursive neural nets with back propagation" you might consider the approach used by Frantzi, et. al. at National Centre for Text Mining (NaCTeM) at University of Manchester for Termine (see: http://www.nactem.ac.uk/index.php and http://personalpages.manchester.ac.uk/staff/sophia.ananiadou/IJODL2000.pdf) Instead of deep neural nets, they "combine linguistic and statistical information".

Two recent papers use a Deep learning architecture called CharWNN to address this problem. CharWNN was first used to get state of the art results (without handcrafted features) on Part of Speech (POS) tagging on an English corpus.

The second paper by the same author uses the same (or similar) architecture for predicting whether a word belongs to 10 Named Entity classes, with apparent state of the art results.

• Interesting. Thanks for sharing this. I wonder why they didn't test this approach on english corpora. – MaticDiba Sep 14 '15 at 11:33

Try http://deeplearning4j.org/word2vec.html. This has an implementation of Word2Vec used instead of Bag of Words for NER and other NLP tasks.

• I've had no problem finding word2vec implementations, but I have been unable to find a working recursive net to use. – Madison May Aug 28 '14 at 16:57
• Link is no more active, if possible kindly share the new working link – Amandeep Mar 15 at 10:51

Here are some ideas about how to use word vectors for NER, which adopts a mostly unsupervised word2vec-centric approach.

1. Given a set of word vectors (either something you trained, or something off-the-shelf like GoogleNews-vectors-negative300.bin), discover clusters in the vector space. These clusters are basically your definitions for various unnamed concepts.
2. With minimal supervision, you can map/convert the unnamed clusters to match human knowledge, thus creating named concepts grounded on top of the known word vectors and unnamed concepts. For example, a method findCluster(['joy', 'surprise', 'disgust', 'trust', 'fear', 'sadness', 'anger', 'anticipation']) might return a list containing hundreds of words mostly related to emotion. If you name this list 'emotion', then there you have a named concept 'emotion' defined based on the vector space.
3. You can also do vector math to find the intermediate concept between two given ones. For example, the vector math might tell you that when given two words 'surprise' and 'disgust', the following are found between the two: dismay, astonishment, disappointment, amazement, bewilderment, exasperation, disbelief, shock, etc. This allows you to build up relationship between concepts.
4. You can repeat the above to build up various types of named concpts, like: weekdays, all emotions, happy emotions, vehicles, etc.
5. Once you have built up layers of named concepts, you can then train a RNN on a text corpus that has been augmented with the named concepts, so 'brown fox jumps' is also '{color} {animal} {action}' etc. This way the RNN should be able to learn some rudimentary grammar in an unsupervised manner.
6. If you have built up a sufficiently powerful grammar from the above, then you should be able to apply it toward some of your NER tasks.