0
$\begingroup$
  1. If no, what are the other criteria. Please elaborate.

  2. What should be minimum value of AUC to select a model.

| improve this question | | | | |
$\endgroup$
1
$\begingroup$

Not at all, and AUC is not a particularly well-respected measure of model performance. It performs particularly poorly for rare events, since the usual criterion for "best fit" is obtained for sensitivities between .7 and .9 where the specificities will typically be also .7 to .9 for evan a good model. The choice of a proper "best" criterion will very much depend on the type of outcome, the frequency of the outcomes of interest and any weighting or costs associated with them. Naive application of the AUC will have no consideration of the cost of false positives or false positives

You should read up on calibration and cross-validation. This is quite a broad topic, but the statisticians have been at it for a while and there is a lot to be learned by searching an affiliated site, CrossValidated.com: https://stats.stackexchange.com/search?q=best+fit+cross-validation

The minimum AUC would of course be 0.5, which is what you would get from pure chance. There is no established minimum. Even teh p < 0.05 is an aribitrary boundary, not at all established by a theory. You need to decide how much to value good versus bad decisions from the task at hand.

| improve this answer | | | | |
$\endgroup$
3
$\begingroup$

No, area under receiver operating characteristic (AUROC) is just one metric amongst very many possibilities, even assuming you just want to pick a standard approach. There are too many to list in a simple Stack Exchange answer. You can take a look at this list extracted from scikit learn documentantion on metrics for example, which is not by any means exhaustive:

Accuracy classification score.
Area Under the Curve (AUC) using the trapezoidal rule
Average precision (AP) from prediction scores
The Brier score.
The F1 score, also known as balanced F-score or F-measure
The F-beta score
The average Hamming loss.
Average hinge loss (non-regularized)
Jaccard similarity coefficient score
Log loss, aka logistic loss or cross-entropy loss.
The Matthews correlation coefficient (MCC) for binary classes
Area Under the Curve (AUC) from prediction scores
Zero-one classification loss.

What you should choose depends entirely on your model class and goals for the learning work. AUROC is a good choice for a binary classifier when you have different business costs associated with false positives and false negatives. That is because it gives you a sense of how well the classifier can be tuned to be more or less sensitive, and get the best outcomes by changing the class threshold.

There is no minimum AUC or other metric required to select a model in practice. It depends on performance of existing solutions (including non-ML ones) and what the costs vs benefits would be of using the model. Clearly a poorly-performing model (e.g. with AUROC 0.5, no better than random guessing on a balanced data set) is unlikely to gain much from being implemented in a production environment, and you would want to take a new look at the original problem and see if was reasonable to expect anything from ML at all.

To decide which metric to use, you need to define some goal for the eventual model. An ideal metric is one that can be simply converted to terms that the end users will care about. Failing that, one that can be compared to known results from other ML work in the subject area (domain) of the problem.

| improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.