What are the best method/library/data available to extract named entities [Names and Location] from Twitter data ? [Other than dictionary lookup]

I tried with Python-Stanford NER, But it seems to fail when named entities is not capitalized.

I also tried to predict NER after converting text to upper case eg :

 text = "david beckham played for england"

 [(u'david', u'PERSON'), (u'beckham', u'PERSON'), (u'played', u'O'), (u'for', u'O'), (u'england', u'O')]

 output : [(u'DAVID', u'PERSON'), (u'BECKHAM', u'PERSON'), (u'PLAYED', u'O'), (u'FOR', u'O'), (u'ENGLAND', u'LOCATION')]

Yes it is a challenging task to extract named entities in tweets. Give a go at NLTK NER and also Alan Ritter's Twitter specific NER and evaluate on their performance and compare to Stanford NER and which one fits in your use. Maybe you want to use more than one to get more named entities if you don't mind so much of false NEs..


I think you are in a better position to train your own NER model. You can start with CRFSuite as a package.

  • $\begingroup$ yes preparing model with CRF is ok. But I would like to know is there any other packages/libraries available? like Stanford NER for normal text message. $\endgroup$ – Sreejithc321 Jun 20 '15 at 16:57
  • $\begingroup$ Well the thing is that for open texts the standard CRF models might work poorly. The advantage of having your own is that you can make corrections and re-train. $\endgroup$ – Vladislavs Dovgalecs Jun 20 '15 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.