0
$\begingroup$

I have a dataset with 837377 observations (51% to train, 25% to validation and 24% to test) and 19 features.

I calculated the recall score using average macro for train, validation and test and obtained:

Train: 0.9981845060159042 Val: 0.7559011239753489 Test: 0.7325217067167821

Can I say my multiclass and imbalanced Random Forest model is overfitting by saying that recall_train > recall_val and recall_train > recall_test? Is recall the best metric to use in this case?

$\endgroup$
5
  • $\begingroup$ How many classes has your dataset? What is their distribution? $\endgroup$
    – Eduard
    Commented Feb 8, 2023 at 12:50
  • $\begingroup$ 11 Classes. For the train dataset - 0: 65295, 1: 870, 2: 469, 3: 1943, 4: 100725, ... $\endgroup$ Commented Feb 8, 2023 at 14:40
  • $\begingroup$ BTW, I am also using IoU (Intersection over Union) for this analysis. Maybe this is a better metric in this case. $\endgroup$ Commented Feb 8, 2023 at 17:30
  • $\begingroup$ I do not have practice with IoU, but I have learned that essentially is a fraction $\frac{|A \cap B|}{|A \cup B|}$ where $A$ and $B$ are, for example, geometrical objects (e.g., rectangles). Is this really what you want? $\endgroup$
    – Eduard
    Commented Feb 9, 2023 at 19:07
  • $\begingroup$ Yes, I saw that IoU can also be applied on my case. The equation is the following: IoU = true_positive/(true_positive+false_positive+false_negative). $\endgroup$ Commented Feb 10, 2023 at 12:06

1 Answer 1

0
$\begingroup$

I suggest using the micro or macro F1 score for unbalanced problems like yours.

To understand the difference between micro versus macro metric, read this great answer (and follow-up comments).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.