Metrics to measure imbalanced multi-class problem

Question

I have a multiclass imbalanced problem. The dependent variable is shown below.

              Injury
  severe Injury (Class 0)               3394
  Medium Injury (Class 1)              22318
  No Injury (Class 2)                 208753

I have used the random forest algorithm with parameter class_weight='balanced' to handle the imbalance problem. The model yields the below results.

 [[  650    12     9]
  [    3  2938  1670]
  [    7   917 40569]]

  Accuracy Score: 0.9440299305184393
  precision score: 0.9016230160324789
  Recall score: 0.8612021971135553
  AUC Score: 0.8739141097167544
  F1 score: 0.879571098748252

               precision    recall  f1-score   support

    class 0       0.98      0.97      0.98       671
    class 1       0.76      0.64      0.69      4611
    class 2       0.96      0.98      0.97     41493

avg / total       0.94      0.94      0.94     46775

Kappa Score: 0.7391731672532447

Since I am really interested in class 0 and class 1, the precision, recall, and F1 scores are calculated using "Macro" method.

 ex:
 print ('precision score:', precision_score(test_y,ry_pred, average='macro'))

So, my question is Can I take the individual classes (class 0 and class 1) precision, recall, and f1 ? or the overall average(including class 2) score for evaluating the model?

 ex:  F1 score for all three classes = (98+69+97)/3 = 0.8795 
      F1 Score for 2 classes = (98+69)/2 = 83.5

If I have to take individual classes scores then, can I take class 0 and class 1 aggregate values to say how well the model is classifying the minor classes?

Also, for class 0 the model is showing high scores around 98. Is this an overfitting problem? I have evaluating model using k fold CV technique and shows below scores. It is showing the overall precision, recall, and F1 score.

     precision score 0.9009562240704383
        recall score 0.8576816035552879
            F1 score 0.8776626709718627

Please suggest.

Answer 1

I see two maybe 3 options here.

1.) Don't use class_weight = 'balanced'.

Instead use class_weight = {0 : 0.01447 , 1 : 0.095 , 2 : .890}

2.) over sample the other classes using some like SMOTE.

3.) (possibly) use a model to classify label 2 only in a one-vs-rest scheme, then use a second model classify label 1 or 0. Kinda in a stacked approach. Becareful not to leak information when training though.