I'm trying to extract percentages using Stanford NER. But it is not extracting percentage properly.

import os
from nltk.tag import StanfordNERTagger
os.environ['CLASSPATH'] = 'stanford-ner-2015-12-09/stanford-ner.jar'
os.environ['STANFORD_MODELS'] = 'stanford-ner-2015-12-09/classifiers'

inp_str = 'total revenue received was one hundred and twenty five percent 125% for last financial year'
split_inp_str = inp_str.split()
st = StanfordNERTagger('english.muc.7class.distsim.crf.ser.gz')

This gives following output

[('total', 'O'), ('revenue', 'O'), ('received', 'O'), ('was', 'O'), ('one', 'O'), ('hundred', 'O'), ('and', 'O'), ('twenty', 'O'), ('five', 'PERCENT'), ('percent', 'PERCENT'), ('125%', 'O'), ('for', 'O'), ('last', 'O'), ('financial', 'O'), ('year', 'O')]

Expected ouput:

[('total', 'O'), ('revenue', 'O'), ('received', 'O'), ('was', 'O'), ('one', 'PERCENT'), ('hundred', 'PERCENT'), ('and', 'PERCENT'), ('twenty', 'PERCENT'), ('five', 'PERCENT'), ('percent', 'PERCENT'),('125%', 'PERCENT'), ('for', 'O'), ('last', 'O'), ('financial', 'O'), ('year', 'O')]

Why is it not extracting 125% or one hundred and twenty five percent?

  • $\begingroup$ welcome to data science SE. Could you copy paste a full working example and your expected output? $\endgroup$ – Stereo Jan 9 '17 at 16:54
  • $\begingroup$ @Stereo Thanks . I have added working code as well as expected output. $\endgroup$ – Khaleeque Ansari Jan 10 '17 at 6:34
  • $\begingroup$ It is not predicting the percentage entity every time? $\endgroup$ – Himanshu Rai Jan 10 '17 at 7:05
  • $\begingroup$ @HimanshuRai The output is coming as five percent instead of expected output $\endgroup$ – Khaleeque Ansari Jan 10 '17 at 7:11
  • $\begingroup$ No, I mean did you try for other examples or this is the only sample you tried it for? $\endgroup$ – Himanshu Rai Jan 10 '17 at 7:12

Looking at your code, I dont believe there is any mistake in it. The Stanford NER tagger is modelled by a Conditional Random Field Classifier and is hence bound to make mistakes. If you are able to check for a large number of samples and calculate an accuracy metric on the basis of that you might get a good idea if the pre-trained model is useful for you. Else, I suggest that you train your own model using. The stanford NLP group provide an implementation for training, you just have to give it the data in the required format. Look at this blog to get a good idea on how to do it.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.