4
$\begingroup$

I have built a model using the xgboost package (in R), my data is unbalanced (5000 positives vs 95000 negatives), with a binary classification output (0,1).

I have performed cross validation with the evaluation metric AUC Area under the ROC curve which I now believe to be wrong since this is better used for balanced data sets.

I analyse the final results of the model using the Area Under the Precision Recal curve (AUPRC) and the Matthews Correlation Coefficient (MCC), however I now believe that I should have been evaluating the cross validation models with the AUPRC and MCC also and completely forget about the AUROC.

I cannot find much in the literature which uses CV with the evaluation metric of AUPRC and MCC.

I just want to make sure that I am thinking correctly and that my previous evaluation method is wrong and the AUPRC / MCC would be a better way to go.

Improve this question
$\endgroup$
14
  • 5
    $\begingroup$ It is incorrect that AUC has any issue with class balance. AUC is insensitive to class balance, being the probability that the model scores a positively labeled class higher than a negatively labeled class. You should also consider evaluating model performance with log-loss, which, being based on probabilities, is also insensitive to class balance issues. It's important to seperate evaluating your model from evaluating your decision rule, these are different concerns. $\endgroup$
    – Matthew Drury
    Jun 14, 2018 at 21:09
  • 3
    $\begingroup$ This all depends on your application. What are you building the model for? Are you trying to build a decision rule, that needs to assign classes to new data points, or are the predicted probabilities sufficient for your purposes? Are you more interested in ranking? By evaluating your model with AUC, you are essentially saying that your dominant concern is how well your model ranks the data. The answers to these types of questions governs how you should evaluate your model. If you don't know, a very good default is log loss, since this ensures that your probabilities are accurate. $\endgroup$
    – Matthew Drury
    Jun 14, 2018 at 21:17
  • 1
    $\begingroup$ If the predicted probabilities are sufficient, then you're not building a decision rule, so there is no concept of type one error. If you'd like to ensure that the predicted probabilities are well calibrated to your data, then evaluating and comparing competing models on the basis of log loss is the way to go. You may also want to stratify your data when evaluating to make sure that your model is not just predicting well on some class off applicants (i.e. a model that does well on adults, but is crap on college students). $\endgroup$
    – Matthew Drury
    Jun 14, 2018 at 21:32
  • 1
    $\begingroup$ Note that by drawing something like a PR curve, you're drawing out how the model would preform as a decision rule over an entire range of thresholds. If you have no intention to set a threshold and use the model as a decision rule, than these kinds of evaluations are inappropriate for the application. $\endgroup$
    – Matthew Drury
    Jun 14, 2018 at 21:33
  • 2
    $\begingroup$ Please, please, please don't use 0.5 as a default threshold. This is almost always wrong. If you intend to do this, you NEED to tune the threshold of your model based on your business objectives. The correct way to do this is to estimate the costs to your business of false positives and false negatives, and then use those to set a correct classification threshold. There is really no justification for using 0.5 as a default threshold in practice. $\endgroup$
    – Matthew Drury
    Jun 14, 2018 at 21:40

1 Answer 1

Reset to default
4
$\begingroup$

I feel you are mixing few things:

  1. 1:20 is not exactly a class imbalanced data. Classification algorithms work well with this ration also. If you still feel so, you can oversample your positive class or undersample negative class.
  2. AUROC generally to compare different models. Like one from logistic and other from Xgboost.( It also ensures robustness of models), Generally AUROC is not used for model accuracy/ getting cut-off values.
  3. Logloss/ misclassification rate/ F1 score/ MCC any of these can be used based on your requirement of classification. If you are predicted cancer , then you might not want to miss any patient and might be Ok to false positive, then focus should be on recall. f1 gives balance of precision and recall. Log loss is used (Log Loss heavily penaltises classifiers that are confident about an incorrect classification) when you don't want to have false positive. These links might help.

https://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/

https://www.r-bloggers.com/making-sense-of-logarithmic-loss/

https://towardsdatascience.com/evaluation-metrics-for-classification-409568938a7d

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.