0
$\begingroup$

Hello everyone i'm new to data science world. So i want to know if my model is overfitting. Usually i'm comparing training accuracy and testing accuracy. But on some reference many people using roc_auc score from training and testing and compared it to know if the model is overfit.

Which metrics evaluation better? I have imbalanced data, but already using SMOTE oversampling.

The second question i want to ask if i'm decided to use accuracy method to know if my model is overfitting, should i try using k cross validation with scoring='accuracy' to prove more?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

Indeed as you suggest, ROC AUC is a more robust metric than accuracy. It is even more relevant for imbalanced datasets (very frequent in real use cases) where the model might be very good at predicting the majority class but not the minority (which is usually what you are more interested in predicting).

In this case, high accuracy values are misleading when correctly predicting true negatives, but not true positives (minority). What you need is a metric which focuses on getting good values of true positive rates (TPR), and this is something which ROC AUC considers (giving you also a value of how the model behaves considering several model thresholds):

enter image description here (source)

Even a better metric for imbalanced datasets can be the Precision-Recall AUC, you can find here a detailed comparison.

About using cross-validation, yes, you should always try to apply this technique, to prevent overfitting to a unique test set; this gives you robustness since your metric is evaluated on several k-evaluation sets (source).

Improve this answer
$\endgroup$
2
  • $\begingroup$ thank you for your answer, it help me to understand more. But i have one more question.. If i've already done oversampling for my training data, can i use accuracy instead of roc auc? Because from what you've state above that ''Indeed as you suggest, ROC AUC is a more robust metric than accuracy. It is even more relevant for imbalanced datasets (very frequent in real use cases) where the model might be very good at predicting the majority class but not the minority (which is usually what you are more interested in predicting)." $\endgroup$
    – Jovian Aditya
    Oct 12, 2022 at 2:25
  • $\begingroup$ I would first try to train the model using ROC AUC and not using oversampling; actually, I would rather use undersampling instead of oversampling in case the first approach is not good enough (you can try both ways) $\endgroup$
    – German C M
    Oct 12, 2022 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.