I am performing Named Entity Recognition using Stanford NER. I have successfully trained and tested my model. Now I want to know:
1) What is the general way of measuring accuracy of NER model ?? For example what techniques or approaches are used ??
2) Is there any built-in method in STANFORD NER for evaluating the accuracy ??
http://en.wikipedia.org/wiki/Named-entity_recognition#Formal_evaluation :
To evaluate the quality of a NER system's output, several measures have been defined. While accuracy on the token level is one possibility, it suffers from two problems: the vast majority of tokens in real-world text are not part of entity names as usually defined, so the baseline accuracy (always predict "not an entity") is extravagantly high, typically >90%; and mispredicting the full span of an entity name is not properly penalized (finding only a person's first name when their last name follows is scored as ½ accuracy).
In academic conferences such as CoNLL, a variant of the F1 score has been defined as follows:
- Precision is the number of predicted entity name spans that line up exactly with spans in the gold standard evaluation data. I.e. when [Person Hans] [Person Blick] is predicted but [Person Hans Blick] was required, precision for the predicted name is zero. Precision is then averaged over all predicted entity names.
- Recall is similarly the number of names in the gold standard that appear at exactly the same location in the predictions.
- F1 score is the harmonic mean of these two.
It follows from the above definition that any prediction that misses a single token, includes a spurious token, or has the wrong class, "scores no points", i.e. does not contribute to either precision or recall.
Given that you run a test against labeled data (the -testFile option), then Stanford NER will report entity level F1 scores, and a micro-average entity-level F1 at the end of the run.
