1
$\begingroup$

I work on a dataset of 300 000 samples and I try to make a comparison between logistic regression (with gradients descent) and a LightBoost for binary classification in order to choose the better one.

I want to know in this case which metric should I use it and WHY?

  • Accuracy ??

  • AUC Test value ??

  • RMSE ??

  • LogLoss ??

$\endgroup$
2
  • $\begingroup$ Please come over to Cross Validated, stats.stackexchange.com, and look for proper scoring rules. Accuracy and AUC are not proper scoring rules, while RMSE (square root of Brier score in a classifier) and log loss are proper scoring rules. Shamelessly, I will link a question of mine that has an excellent answer about proper scoring rules and links to other good content: stats.stackexchange.com/questions/464636/…. $\endgroup$ – Dave Jun 15 '20 at 3:27
  • $\begingroup$ The “Frank Harrell” (Vanderbilt Star Professor) I mention has blog posts about proper scoring rules, too: fharrell.com/post/class-damage and fharrell.com/post/classification. He also posts about proper scoring rules on Cross Validated and has addressed why even AUC is not viable as a metric for comparing models, though one of his posts mentions that AUC is fine as a diagnostic for a single mode to check it it’s doing a decent job (since log loss can be tough to interpret, but 0.99 AUC, for example, sounds pretty good). $\endgroup$ – Dave Jun 15 '20 at 3:31
5
$\begingroup$

Depends.

The first thing that has to be clear is that you are running an experiment, which means you need to measure both with the same metric.

Which one? Depends on which underlying problem you are solving, if what you are doing is to determine which algorithm is better, your conclusion will only be applicable to your specific dataset

  • Accuracy: Is possible to measure the accuracy as the comparing metric, but it becomes trivial if your dataset is unbalanced, which means you have much more positives than negatives or viceversa. The accuracy is used when the dataset is balanced and if is equally bad to have a mistake on positives and negatives. Also, it has the problem of being too dependant on the threshold for defining positives/negatives.

  • Area Under the Curve: The AUC is one of the most robust metrics for measuring the ability of a model to separate positives and negatives, is unsensitive to threshold and immune to unbalanceness. I would use this.

  • RMSE: I only know RMSE for continuous regression, not for classification.

  • LogLoss: Its use is in multinomial classification

$\endgroup$
3
  • $\begingroup$ Doesn't LogLoss give a fair comparison between the overall performance of two probabilistic classifiers that is unrelated to threshold / data imbalance? I don't think it suffers from the same problems as accuracy, but would be keen to know if I have missed a detail. $\endgroup$ – ajrwhite Apr 16 '19 at 14:50
  • $\begingroup$ Thank you Juan for your answer , so as conclusion because I work on umbalanced data I should just compare both of them based on AUC $\endgroup$ – Nirmine Apr 16 '19 at 15:13
  • $\begingroup$ You are right @ajrwhite, I will correct my answer $\endgroup$ – Juan Esteban de la Calle Apr 16 '19 at 15:23
3
$\begingroup$

I would say AUC is the best overall metric for classification but does not have to be the only metric, accuracy is useful too. For reference you can check this Quora regarding accuracy vs. AUC:

They both measure different things, so they are complementary.

Accuracy: Measures, for a given threshold, the percentage of points correctly classified, regardless of which class they belong to.

AUC: Measures the likelihood that given two random points — one from the positive and one from the negative class — the classifier will rank the point from the positive class higher than the one from the negative one (it measures the performance of the ranking really).

Log loss can also be a good candidate as an overall metric, why can be read here from FastAI:

Log Loss vs Accuracy

Accuracy is the count of predictions where your predicted value equals the actual value. Accuracy is not always a good indicator because of its yes or no nature.

Log Loss takes into account the uncertainty of your prediction based on how much it varies from the actual label. This gives us a more nuanced view into the performance of our model.

RMSE on the other hand is a regression metric and should not be used for classification.

$\endgroup$
2
  • 1
    $\begingroup$ I fail to see the difference between optimization metric and "evaluation metric". There may be statistics that are well suited for validation/evaluation because they can be adapted for the train/test methodlogy but there is no reason to believe that log loss cannot be used to evaluate a classifier. In fact, log loss is incredibly popular in a large amount of Kaggle competitions, as well as the Brier score. Another similar metric: RMSE. RMSE is optimized in countless regression algorithms to derive parameter estimates/splits but is also used to evaluate the performance of a regression model. $\endgroup$ – aranglol Apr 16 '19 at 20:38
  • 2
    $\begingroup$ That is because I was wrong. I was under the faulty impression that log loss was rarely used for evaluation and was mostly used as a loss function. Thank you for fact checking, will edit to avoid misleading other! :) $\endgroup$ – Simon Larsson Apr 16 '19 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.