2
$\begingroup$

Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime as this always leads to high values due to a number of O (other) tags in the dataset.

$\endgroup$
1
$\begingroup$

As a first start I would recommend to use Precision and Recall.

You can also produce stats on the different types of errors:

  • entity not found (false negative, or Type I error). Also you can get the false negative rate by dividing by the total number of true entities. This would be 1-recall.

  • entity found where it shouldn't be (false positive or Type II error). Also you can get the false positive rate by dividing by the total number of returned entities. This would be 1-precision.

  • entity found but system got wrong number of tokens (e.g. York instead of New York). You can also get the "token boundary error rate" by dividing by the total number of true entities.

So the first two error rates I mentioned don't give any extra info other than precision and recall but the third allows you to check how accurately you are getting entity boundaries.

If you are doing entity disambiguation too, you can produce additional metrics by only counting a success if the entity was correctly resolved.

You can also use the F-score (thanks Erwan) which combines precision and recall. However you need to be careful because you have a large class imbalance of O-tokens vs entity tokens, and the F-score gives equal importance to precision and recall. You also have the options of weighted F-scores such as F2 score.

Other option: ROC curve and AUC

Finally I would add that if your model outputs a score or probability rather than a binary classification, it means you can tweak its sensitivity by changing the threshold (do you want to count all entities with score>0.5 as entities, or score>0.75, for example). This naturally leads to the possibility of calculating the false positive and false negative rates for every possible threshold choice. You can plot it on a graph and you have a ROC curve. This is a nice graphical way to see the performance of your model and exactly how it misclassifies instances. The area under the ROC curve is itself a nice metric called AUC. A classifier that is as good as random has AUC=0.5, a classifier that classifies everything wrong has AUC=0, and a perfect classifier has AUC=1. I normally plot the ROC curve as well as generating metrics, Sklearn and most stats packages have a function to do it.

$\endgroup$
  • $\begingroup$ Excellent answer, I would just add that in case a single evaluation metric is required, the F-score is a standard way to combine precision and recall. $\endgroup$ – Erwan Jun 13 at 14:36
  • $\begingroup$ Thanks, good point. I will edit the answer $\endgroup$ – Tom Jun 13 at 15:00
0
$\begingroup$

If you are using a single evaluation metric like F-score, you can exclude the high frequency label while calculating the metrics If you are using sklearn to calculate the metrics, you would want to use the labels property where you can set the classes on which you want to calculate the metrics.

More information here:

https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html

New contributor
Praneeth is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.