Which metrics that can be use to know "overfitting" model"?

Question

Hello everyone i'm new to data science world. So i want to know if my model is overfitting. Usually i'm comparing training accuracy and testing accuracy. But on some reference many people using roc_auc score from training and testing and compared it to know if the model is overfit.

Which metrics evaluation better? I have imbalanced data, but already using SMOTE oversampling.

The second question i want to ask if i'm decided to use accuracy method to know if my model is overfitting, should i try using k cross validation with scoring='accuracy' to prove more?

Answer 1

Indeed as you suggest, ROC AUC is a more robust metric than accuracy. It is even more relevant for imbalanced datasets (very frequent in real use cases) where the model might be very good at predicting the majority class but not the minority (which is usually what you are more interested in predicting).

In this case, high accuracy values are misleading when correctly predicting true negatives, but not true positives (minority). What you need is a metric which focuses on getting good values of true positive rates (TPR), and this is something which ROC AUC considers (giving you also a value of how the model behaves considering several model thresholds):

(source)

Even a better metric for imbalanced datasets can be the Precision-Recall AUC, you can find here a detailed comparison.

About using cross-validation, yes, you should always try to apply this technique, to prevent overfitting to a unique test set; this gives you robustness since your metric is evaluated on several k-evaluation sets (source).