I am working on a problem on Named Entity Recognition. Given a text, my model is detecting the Named Entities and extracting that info for the end-user. Now the ask is end-user needs a confidence score along with the extracted entity. For example, the given text is: XYZ Bank India Limited is a good place to invest your money - Our model is detecting XYZ Bank as an Org, but India as a Location (which is wrong - the whole XYZ Bank India Limited is the name of the organization). Our model also gives a probability score for each token it classifies. But the end-user wants to know the confidence of the model that it did not mistake to detect the subsequent tokens as the parts of the organization name.

Question is - how can we efficiently measure that in a given sequence our model is detecting a certain sub-sequence as an Organization name (or a Location or something else) correctly or not? How can we say that it did not miss out on any subsequent or preceding token which actually a part of the named entity (like it missed India Limited in the above example)?


Named Entity Recognition is traditionally evaluated using precision/recall and F1 score [https://towardsdatascience.com/entity-level-evaluation-for-ner-task-c21fb3a8edf] - the medium article gives a low down on how to achieve this I recently happened to read this article on a new approach for the same. Please see the details in the attached medium link : [https://towardsdatascience.com/a-pathbreaking-evaluation-technique-for-named-entity-recognition-ner-93da4406930c] but havent tried this out yet though

  • $\begingroup$ Thanks, Vivek. But my intention is not actually to derive an evaluation metric - but it is to communicate end user (who will use the model in production) that for that particular prediction task what is the confidence score. $\endgroup$ May 12 at 6:19
  • $\begingroup$ Ok @SaikatBhattacharya - are you using spaCy NER or Stanford NER package ? I believe both of these provides confidence scores out of the box $\endgroup$
    – vivek
    May 12 at 16:28
  • $\begingroup$ Vivek, we are using sklearn-crfsuit alongwith Fasttext embedding $\endgroup$ May 17 at 5:36
  • $\begingroup$ Sorry for the delay @SaikatBhattacharya - i havent used CRF and preferred spaCy or LSTM for custom entity extraction. Please check the documentation of crfsuite - [sklearn-crfsuite.readthedocs.io/en/latest/_modules/… in here there is a function which returns predicted proba which I believe can be used confidence score when communicating to end user $\endgroup$
    – vivek
    May 24 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.