1
$\begingroup$

I am working on a binary classification using random forest with 977 records with 77:23 class proportion

I got the below performance in train and test data (AUC = 81)

Train data

enter image description here

Test data

enter image description here

My metric of interest is F1-score because my dataset is imbalanced.

So, based on the above classification report, we can infer that drop in F1-score is not huge between train and test. Am I right to understand that?

How much is considered a huge drop?

Should I be worried about Precision and Recall values being dropped heavily? I don't assess my model based on precision and recall indvidually. Should I just focus on F1 and conclude whether my model is overfit or not? Or all the metrics should have some decent/gradual drop?

If we see 1.00 for any of the metric, is it sure shot way to say that model is overfitting?

update - new run screnshot - 7 point drop in f1-score

enter image description here

$\endgroup$

1 Answer 1

2
$\begingroup$

When we talk about huge drop, we do not have thumb rule that this is huge. Anything =>5% should raise eyebrows and can indicate overfitting.

It depends on the business problem you are trying to solve. If precision and recall goes down F1 score will go down. In this case i can clearly see for class1 which is class of interest f1score is going down from 0.72 to 0.62 and also classification metrics is worse than train data. I think this is a case of Overfitting.

You may be looking at wrong f1-score please check. You look for f1-score of minority class as it is class of interest not majority class.

$\endgroup$
2
  • $\begingroup$ thanks. upvoted.. Yes, f1-score for class of interest (class 1) is down by 10 points. agree. but am using only 6 features (parsimonius model) from 62 features. even then if it is overfitting (as you say), what can be done to solve this problem. Should I pick a model where the performance drop is only <=5% (for whichever metric we choose to focus on)? $\endgroup$
    – The Great
    Mar 28, 2022 at 6:17
  • $\begingroup$ I added a new screenshot to my post. Would you say that even this new execution is overfitting? $\endgroup$
    – The Great
    Mar 28, 2022 at 6:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.