I'm a PhD student applying ML in microbiology. In research papers, the usual performance measure reported on classification models is ROC-AUC. But when I look at implementations, the scoring function in the cross-validation is always left default, which results in accuracy. What is the point of running a cross-validation with accuracy and then reporting ROC-AUC on the test set? What am I missing here?
1 Answer
$\begingroup$
$\endgroup$
It does sound a bit inconsistent to me, and in usual circumstances I'd probably opt for a different strategy.
Tuning the model for accuracy suggests that accuracy is the metric of importance for the deployed model. But if you then define the final performance measure as $AUC_{ROC}$, it suggests that performance is characterised primarily by the model's discriminative ability, which begs the question as to why the model was tuned for accuracy specifically.
If anything, I think they should be the other way round. Tune the model for AUC, and report the final score using accuracy. That way you incentivise the tuning for accuracy via a more precise metric like the AUC, whilst also being able to report an intuitive and easy-to-interpret accuracy score for the final model.
It could be that the authors initially tuned and tested the models for accuracy (it's an easy metric to understand and navigate), but the paper's reviewers requested they report AUC scores. In that case, the authors could have simply run the scoring again to get the AUC scores on the old models. This doesn't invalidate the models in my view, it just means that the tuning could perhaps be more optimal. They could report "models were tuned for accuracy and we report final test performance using $AUC_{ROC}$".
I suppose tuning for accuracy could also act as a regulariser compared to AUC, since the former is likely less sensitive to the exact probabilities a model renders, and may impart better generalisation behaviour. If the authors were emphatic about the need for model regularisation, or in your field overfitting and small sample sizes are common, that might have been one of the strategies taken.
I think it's worth asking the authors themselves as they'll know their particular dataset well. Ultimately, they were aiming for a good model, so they will likely have specific reasoning for the point you've raised.
