4
$\begingroup$

What are the best method/library/data available to extract named entities [Names and Location] from Twitter data ? [Other than dictionary lookup]

I tried with Python-Stanford NER, But it seems to fail when named entities is not capitalized.

I also tried to predict NER after converting text to upper case eg : 

 text = "david beckham played for england"

 stanford.NERTagger.tag(text)
 [(u'david', u'PERSON'), (u'beckham', u'PERSON'), (u'played', u'O'), (u'for', u'O'), (u'england', u'O')]

 stanford.NERTagger.tag(text.upper())
 output : [(u'DAVID', u'PERSON'), (u'BECKHAM', u'PERSON'), (u'PLAYED', u'O'), (u'FOR', u'O'), (u'ENGLAND', u'LOCATION')]
Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

Yes it is a challenging task to extract named entities in tweets. Give a go at NLTK NER and also Alan Ritter's Twitter specific NER and evaluate on their performance and compare to Stanford NER and which one fits in your use. Maybe you want to use more than one to get more named entities if you don't mind so much of false NEs..

Improve this answer
$\endgroup$
1
$\begingroup$

I think you are in a better position to train your own NER model. You can start with CRFSuite as a package.

Improve this answer
$\endgroup$
2
  • $\begingroup$ yes preparing model with CRF is ok. But I would like to know is there any other packages/libraries available? like Stanford NER for normal text message. $\endgroup$
    – Sreejithc321
    Commented Jun 20, 2015 at 16:57
  • $\begingroup$ Well the thing is that for open texts the standard CRF models might work poorly. The advantage of having your own is that you can make corrections and re-train. $\endgroup$
    – Vladislavs Dovgalecs
    Commented Jun 20, 2015 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.