Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime as this always leads to high values due to a number of O (other) tags in the dataset.
As a first start I would recommend to use Precision and Recall.
You can also produce stats on the different types of errors:
entity not found (false negative, or Type I error). Also you can get the false negative rate by dividing by the total number of true entities. This would be 1-recall.
entity found where it shouldn't be (false positive or Type II error). Also you can get the false positive rate by dividing by the total number of returned entities. This would be 1-precision.
entity found but system got wrong number of tokens (e.g. York instead of New York). You can also get the "token boundary error rate" by dividing by the total number of true entities.
So the first two error rates I mentioned don't give any extra info other than precision and recall but the third allows you to check how accurately you are getting entity boundaries.
If you are doing entity disambiguation too, you can produce additional metrics by only counting a success if the entity was correctly resolved.
You can also use the F-score (thanks Erwan) which combines precision and recall. However you need to be careful because you have a large class imbalance of O-tokens vs entity tokens, and the F-score gives equal importance to precision and recall. You also have the options of weighted F-scores such as F2 score.
Other option: ROC curve and AUC
Finally I would add that if your model outputs a score or probability rather than a binary classification, it means you can tweak its sensitivity by changing the threshold (do you want to count all entities with score>0.5 as entities, or score>0.75, for example). This naturally leads to the possibility of calculating the false positive and false negative rates for every possible threshold choice. You can plot it on a graph and you have a ROC curve. This is a nice graphical way to see the performance of your model and exactly how it misclassifies instances. The area under the ROC curve is itself a nice metric called AUC. A classifier that is as good as random has AUC=0.5, a classifier that classifies everything wrong has AUC=0, and a perfect classifier has AUC=1. I normally plot the ROC curve as well as generating metrics, Sklearn and most stats packages have a function to do it.
If you are using a single evaluation metric like F-score, you can exclude the high frequency label while calculating the metrics If you are using sklearn to calculate the metrics, you would want to use the labels property where you can set the classes on which you want to calculate the metrics.
More information here: