I have a multiclass imbalanced problem. The dependent variable is shown below.
Injury
severe Injury (Class 0) 3394
Medium Injury (Class 1) 22318
No Injury (Class 2) 208753
I have used the random forest algorithm with parameter class_weight='balanced' to handle the imbalance problem. The model yields the below results.
[[ 650 12 9]
[ 3 2938 1670]
[ 7 917 40569]]
Accuracy Score: 0.9440299305184393
precision score: 0.9016230160324789
Recall score: 0.8612021971135553
AUC Score: 0.8739141097167544
F1 score: 0.879571098748252
precision recall f1-score support
class 0 0.98 0.97 0.98 671
class 1 0.76 0.64 0.69 4611
class 2 0.96 0.98 0.97 41493
avg / total 0.94 0.94 0.94 46775
Kappa Score: 0.7391731672532447
Since I am really interested in class 0 and class 1, the precision, recall, and F1 scores are calculated using "Macro" method.
ex:
print ('precision score:', precision_score(test_y,ry_pred, average='macro'))
So, my question is Can I take the individual classes (class 0 and class 1) precision, recall, and f1 ? or the overall average(including class 2) score for evaluating the model?
ex: F1 score for all three classes = (98+69+97)/3 = 0.8795
F1 Score for 2 classes = (98+69)/2 = 83.5
If I have to take individual classes scores then, can I take class 0 and class 1 aggregate values to say how well the model is classifying the minor classes?
Also, for class 0 the model is showing high scores around 98. Is this an overfitting problem? I have evaluating model using k fold CV technique and shows below scores. It is showing the overall precision, recall, and F1 score.
precision score 0.9009562240704383
recall score 0.8576816035552879
F1 score 0.8776626709718627
Please suggest.