2
$\begingroup$

I have a multiclass imbalanced problem. The dependent variable is shown below.

              Injury
  severe Injury (Class 0)               3394
  Medium Injury (Class 1)              22318
  No Injury (Class 2)                 208753

I have used the random forest algorithm with parameter class_weight='balanced' to handle the imbalance problem. The model yields the below results.

 [[  650    12     9]
  [    3  2938  1670]
  [    7   917 40569]]

  Accuracy Score: 0.9440299305184393
  precision score: 0.9016230160324789
  Recall score: 0.8612021971135553
  AUC Score: 0.8739141097167544
  F1 score: 0.879571098748252

               precision    recall  f1-score   support

    class 0       0.98      0.97      0.98       671
    class 1       0.76      0.64      0.69      4611
    class 2       0.96      0.98      0.97     41493

avg / total       0.94      0.94      0.94     46775

Kappa Score: 0.7391731672532447

Since I am really interested in class 0 and class 1, the precision, recall, and F1 scores are calculated using "Macro" method.

 ex:
 print ('precision score:', precision_score(test_y,ry_pred, average='macro'))

So, my question is Can I take the individual classes (class 0 and class 1) precision, recall, and f1 ? or the overall average(including class 2) score for evaluating the model?

 ex:  F1 score for all three classes = (98+69+97)/3 = 0.8795 
      F1 Score for 2 classes = (98+69)/2 = 83.5

If I have to take individual classes scores then, can I take class 0 and class 1 aggregate values to say how well the model is classifying the minor classes?

Also, for class 0 the model is showing high scores around 98. Is this an overfitting problem? I have evaluating model using k fold CV technique and shows below scores. It is showing the overall precision, recall, and F1 score.

     precision score 0.9009562240704383
        recall score 0.8576816035552879
            F1 score 0.8776626709718627

Please suggest.

$\endgroup$
2
$\begingroup$

I see two maybe 3 options here.

1.) Don't use class_weight = 'balanced'.

Instead use class_weight = {0 : 0.01447 , 1 : 0.095 , 2 : .890}

2.) over sample the other classes using some like SMOTE.

3.) (possibly) use a model to classify label 2 only in a one-vs-rest scheme, then use a second model classify label 1 or 0. Kinda in a stacked approach. Becareful not to leak information when training though.

$\endgroup$
  • $\begingroup$ My question is after passing this parameter, class_weight = {0 : 0.01447 , 1 : 0.095 , 2 : .890}, Can I take the overall precision and recall scores for model evaluation? or Shall I take only class 0 and class 1 scores since I am interested in minority classes? $\endgroup$ – Bhaskar Sharma Nov 19 '18 at 12:46
  • $\begingroup$ Hmm I'd still optimize to the minority class scores in that case. The class weights don't change what you optimize to really. $\endgroup$ – ASS466uiuc Nov 19 '18 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.