0
$\begingroup$

My dataset is of shape – 5621*8 (binary classification)

  • Label/target : Success (4324, 77 %) & Not success (1297, 23 %)

    (success and Not success were been equally important for my prediction i.e, TP & TN)

I split my data into 3 (Train, Validate, test)

  • For train & Validate i perform 10 fold CV.

  • Test is the seperate data, which I evaluate for each folds

I tune my scale_pos_weight ranging between 5 to 80, and

  • Finally I fixed my values as 75 since I got average higher accuracy rate for my Test set (79 %) for those 10 folds
  • But, If i check my average auc_roc metrics it is very poor, i.e only 50 % for all 10 folds.

If i did not tune scale_pos_weight my avg.accuracy drops to 50% & my avg auc_roc increases to 70 %.

How can I interpret from the above results between AUCROC & Accuracy in this situation?

What might be the problem in my case?

Improve this question
$\endgroup$
5
  • $\begingroup$ What does "I need both the classes to perform good" mean? Is accuracy or auroc type of metrics more important to the problem you are solving? They measure different aspects of the model. Is the cutoff value used for accuracy the right cutoff value for your problem and this trained model? $\endgroup$
    – Craig
    Commented Jun 18, 2020 at 9:59
  • $\begingroup$ I have updated my question, my both classes has been equally important, whcih means am not concerned about only success nor not success.I require both. Also, related to cut off, I have not modify the value, it is default. $\endgroup$
    – Mari
    Commented Jun 18, 2020 at 10:04
  • $\begingroup$ Which class is the "positive" class? (How are you encoding Success/NotSuccess?) $\endgroup$
    – Ben Reiniger ♦
    Commented Jun 18, 2020 at 15:06
  • $\begingroup$ Success as Positive class (Not success as negative class) $\endgroup$
    – Mari
    Commented Jun 18, 2020 at 15:14
  • $\begingroup$ Is the default cut off the correct cut off for your model? Is the default 0.5? Perhaps, given no other information the cutoff should be the event rate? But it should be researched as part of your model tuning if using this metric. Cost of FP and FN vs benefit of TP and TN. Which concept of the metrics accuracy vs AUROC is more important to your model? They measure different things. You may not be able to optimize both. And sometimes, the model is as good as it can be with the current specification. I do not know in this case. $\endgroup$
    – Craig
    Commented Jun 19, 2020 at 12:51

1 Answer 1

Reset to default
1
$\begingroup$

With Success already being the larger class, you probably shouldn't be using a scale_pos_weight larger than one: you want to scale the positive class's contribution to the loss function down rather than up.

I suspect that's what's happening in the first case. With scale_pos_weight=75, the model ends up basically only caring about the positive class, predicts everyone is in the positive class, and so your accuracy is just a little better than the 77% baseline you'd expect with that strategy. With that motivation, it's not too surprising the AUC is poor, although I wouldn't have expected a drop all the way to the 50% baseline...

If you don't use scale_pos_weight (you said "if I did not tune", but does that mean you left it at the default 1?), then the model performs better in rank-ordering (AUC=70%), but not so well in the hard classification. You might want to tweak the prediction threshold here; there's probably a different threshold that will perform better for accuracy score. You could also try scale_pos_weight=0.25 or so; that should make the default cutoff better, hopefully with little effect on AUC?

Improve this answer
$\endgroup$
1
  • $\begingroup$ you are right, I wrongly misunderstood scale_pos_weight. I thought the classifier would take the lower % class as positive and higher % as negative. But if I input scale_pos_weight = 0.25 my accuracy is not improved, whether i can chnage my label (like 0 to 1 & 1 to 0) and then try with the scale_pos_weight params. I will try it, but what is your opinion ??? $\endgroup$
    – Mari
    Commented Jun 18, 2020 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.