I'm currently using several different classifiers on various entities extracted from text, and using precision/recall as a summary of how well each separate classifier performs across a given dataset.
I'm wondering if there's a meaningful way of comparing the performance of these classifiers in a similar way, but which also takes into account the total numbers of each entity in the test data that's being classified?
Currently, I'm using precision/recall as a measure of performance, so might have something like:
Precision Recall Person classifier 65% 40% Company classifier 98% 90% Cheese classifier 10% 50% Egg classifier 100% 100%
However, the dataset I'm running these on might contain 100k people, 5k companies, 500 cheeses, and 1 egg.
So is there a summary statistic I can add to the above table which also takes into account the total number of each item? Or is there some way of measuring the fact that e.g. 100% prec/rec on the Egg classifier might not be meaningful with only 1 data item?
Let's say we had hundreds of such classifiers, I guess I'm looking for a good way to answer questions like "Which classifiers are underperforming? Which classifiers lack sufficient test data to tell whether they're underperforming?".