# What is an appropriate way to compare classifiers with different sets of classes?

I have three classifiers for language identification:

A: en, de, ru, fr, ij, kl
B: en, de, ru, fr, xy
C: en, de, ru, fr, no, pq, rs


and I have a balanced dataset which matches the classes of A.

What is a fair way to compare those classifiers?

## My thoughts

• Accuracy on classes of A: Not fair, because B can't possible recognize ij, kl and will likely make mistakes due to the fact that it knows language xy. (Similar for C)
• Smallest common subset: Possible, but not so interesting as those are the "easy" classes.
• Precision: Given a language the classifier knows, how often does it actually recognize it?
• Unfair for C, as it has more possibilities to make mistakes. Probably this could be somehow compensated? E.g. if it recognizes a class which is not in A, just the next best is taken (until the correct class is predicted or a wrong class from A is preditected)

There could also be a possibility to use the classifier A in combination with another classifier. The weight / importance (however that is measured) of the contribution of the other classifier is its score.

Are there any publications which do something similar?

• I'd separately report performance figures for each language, because different classifiers will likely perform better on different subsets. Finally, you could think about ensembling to dodge the question. – Emre Aug 16 '17 at 19:06
• @Emre But which performance figures? Even a per-class-precision is unfair if one classifier only distinguishes 10 classes and the other 100. – Martin Thoma Aug 16 '17 at 19:12
• @Emre Yes, ensembling is also what I thought. For example, take an extremely simple model for the set I have data for. Take an rather simple model for combining models predictions. Report this basic ensembles accuracy per classifier. Then B might have a fair chance for the set of A. – Martin Thoma Aug 16 '17 at 19:14
• You can set up a cascade so the wider classifiers can first dismiss the possibility of the language belonging to the rarer classes before running the ensemble on the common classes. Another option is to add an "unknown" class to the smaller classifiers. Just some ideas I'd try... – Emre Aug 16 '17 at 19:31
• @Emre In a first step, I don't want to build a good classifier, I want to evaluate given systems. So changing the classifiers is not helpful. – Martin Thoma Aug 16 '17 at 19:50