Visualizing F-score differences in information extraction

Question

I have several corpora and NLP systems (including a few merge ensembles of output of these systems combined in unions and intersections) with which I have extracted the annotation span sets {(begin, end)} for each corpus across all documents within the corpus and compared the span sets to each corpus's respective gold standard and thus obtained standard measures of F-score, precision and recall.

I am trying to qualitatively assess why certain systems don't perform as well as a particular ensemble combination on F-score, so I figured the easiest way would be to generate precision-recall or ROC curves.

The task is just a simple binary classification: either a span of text is annotated (labeled as 1) or it is not (labeled as 0).

I have numpy vectors of the same length for each document in the corpus for both the system predictions and the gold standard, so I plan on using these for y_true and y_predict when trying to generate my ROC curve.

Is this a good approach to observe the behavior of my F-scores, assuming I plot them all on the same graph? If not, any recommendations for a better approach would be most appreciated.

Answer 1

Yes, I think that's a sound approach and a good way to compare different systems.

A ROC curve comparison is usually more informative than the raw performance scores, but it's still quite general. In case you want to observe even more detail, you could also try to look at specific groups of instances. One way to do that is to count for every instance how many systems correctly classify it: an instance almost always correctly classified is "easy", and conversely an instance which is almost always misclassified is "hard". It's often interesting to look at what happens specifically for the "hard" instances with the different systems. You could take a subset of "hard" instances and calculate the performance or ROC curve only on those, in order to distinguish more precisely the best systems.

For the record, if it makes sense for your task you might also want to consider more flexible scoring methods for text spans: currently it seems that your evaluation considers an answer correct only of the exact span is predicted. You could consider counting the fact that a span is partially correct, for instance by counting the number of tokens correctly annotated.