I have three classifiers for language identification:
A: en, de, ru, fr, ij, kl B: en, de, ru, fr, xy C: en, de, ru, fr, no, pq, rs
and I have a balanced dataset which matches the classes of A.
What is a fair way to compare those classifiers?
- Accuracy on classes of A: Not fair, because B can't possible recognize ij, kl and will likely make mistakes due to the fact that it knows language xy. (Similar for C)
- Smallest common subset: Possible, but not so interesting as those are the "easy" classes.
- Precision: Given a language the classifier knows, how often does it actually recognize it?
- Unfair for C, as it has more possibilities to make mistakes. Probably this could be somehow compensated? E.g. if it recognizes a class which is not in A, just the next best is taken (until the correct class is predicted or a wrong class from A is preditected)
There could also be a possibility to use the classifier A in combination with another classifier. The weight / importance (however that is measured) of the contribution of the other classifier is its score.
Are there any publications which do something similar?