I have the following evaluation metrics on the test set, after running 6 models for a binary classification problem:

  accuracy logloss   AUC
1   19%      0.45   0.54
2   67%      0.62   0.67
3   66%      0.63   0.68
4   67%      0.62   0.66
5   63%      0.61   0.66
6   65%      0.68   0.42

I have the following questions:

  • How can model 1 be the best in terms of logloss (the logloss is the closest to 0) since it performs the worst (in terms of accuracy). What does that mean ?
  • How come does model 6 have lower AUC score than e.g. model 5, when model 6 has better accuracy. What does that mean ?
  • Is there a way to say which of these 6 models is the best ?
  • $\begingroup$ What's the ratio of the classes in your dataset? $\endgroup$ Oct 30 '19 at 2:40
  • $\begingroup$ the ratio is around 1/5 $\endgroup$
    – quant
    Oct 30 '19 at 7:44

I will start with your last question as the answer explains the other questions. In general, this is a hard question. That is exactly why there are different metrics available. Depending on your problem and what you want to achieve, you should choose the metric that measures this best.

The differences in metrics can be found all over the internet. Others have explained it much better than I can.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.