1
$\begingroup$

I have three classifiers for language identification:

A: en, de, ru, fr, ij, kl
B: en, de, ru, fr, xy
C: en, de, ru, fr, no, pq, rs

and I have a balanced dataset which matches the classes of A.

What is a fair way to compare those classifiers?

My thoughts

  • Accuracy on classes of A: Not fair, because B can't possible recognize ij, kl and will likely make mistakes due to the fact that it knows language xy. (Similar for C)
  • Smallest common subset: Possible, but not so interesting as those are the "easy" classes.
  • Precision: Given a language the classifier knows, how often does it actually recognize it?
    • Unfair for C, as it has more possibilities to make mistakes. Probably this could be somehow compensated? E.g. if it recognizes a class which is not in A, just the next best is taken (until the correct class is predicted or a wrong class from A is preditected)

There could also be a possibility to use the classifier A in combination with another classifier. The weight / importance (however that is measured) of the contribution of the other classifier is its score.

Are there any publications which do something similar?

Improve this question
$\endgroup$
5
  • $\begingroup$ I'd separately report performance figures for each language, because different classifiers will likely perform better on different subsets. Finally, you could think about ensembling to dodge the question. $\endgroup$
    – Emre
    Aug 16, 2017 at 19:06
  • $\begingroup$ @Emre But which performance figures? Even a per-class-precision is unfair if one classifier only distinguishes 10 classes and the other 100. $\endgroup$
    – Martin Thoma
    Aug 16, 2017 at 19:12
  • $\begingroup$ @Emre Yes, ensembling is also what I thought. For example, take an extremely simple model for the set I have data for. Take an rather simple model for combining models predictions. Report this basic ensembles accuracy per classifier. Then B might have a fair chance for the set of A. $\endgroup$
    – Martin Thoma
    Aug 16, 2017 at 19:14
  • $\begingroup$ You can set up a cascade so the wider classifiers can first dismiss the possibility of the language belonging to the rarer classes before running the ensemble on the common classes. Another option is to add an "unknown" class to the smaller classifiers. Just some ideas I'd try... $\endgroup$
    – Emre
    Aug 16, 2017 at 19:31
  • $\begingroup$ @Emre In a first step, I don't want to build a good classifier, I want to evaluate given systems. So changing the classifiers is not helpful. $\endgroup$
    – Martin Thoma
    Aug 16, 2017 at 19:50

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.