2
$\begingroup$

I have unbalanced dataset for multiclass classification and I tried to use the class weights option in XGboost and the classifier still tends to favor the majority class. I am not sure if I need to tune something else or how I should approach this. If the algo was predicting all over the place I could still understand but not sure why it still skews towards the majority class. Any pointers?
Update: What I mean is that majority of the predictions are still for the major class which is roughly 30% in both test and train

$\endgroup$
  • $\begingroup$ What do you mean by the classifier still tends to favor the majority class? $\endgroup$ – Franco Piccolo Mar 11 '19 at 14:17
  • $\begingroup$ Sorry if it was not clear..i mean overwhelming majority of the predictions are still for the major class...hope that makes sense $\endgroup$ – Swap Mar 11 '19 at 15:06
2
$\begingroup$

That is completely normal. You should remember that the model will basically learn a statistical function given by your data (Intuitively), and since your data is skewed, it will learn by the majority class.

To overcome that, you can treat the imbalance characteristic of data set using two types of approaches: sampling and cost-sensitive learning. Basically, in sampling techniques we have undersampling and oversampling, that is, we can reduce our data set to balance data or increase it with artificial methods. Some examples:

  • Random Oversampling: essentially, this technique increases minority class by randomly sampling it.
  • SMOTE: random oversampling has a overfitting problem as we are just replicating data considerably. To overcome that, smote works, in a simplified way, interpolating points on the feature space for nearest neighbors in minority classes.

For undersampling techniques:

  • Random Undersampling: this method simple undersample the majority class. We can loss information, but if points are close, it can give good results

  • NearMiss: In a simplified way, this model selects the closest points to the minority class.

In cost-sensitive learning techniques, we use cost matrix 'punishing' models that give wrongs score to a specific class. For example, let's say you want to give better results for class C1, you can create a matrix where the misclassification of C1 is higher than other classes.

In python we have two excellents libraries to deal with this problems:

They have excellent tutorials on the topic so you can get a deeper overview of methods discussed and others ones. I hope it helps.

$\endgroup$
  • $\begingroup$ oh..so if I am understanding correct..the sample_weight attribute for xgboost will not necessarily handle the imbalance? I had tried SMOTE...let me explore teh cost classification route.. $\endgroup$ – Swap Mar 11 '19 at 17:21
  • $\begingroup$ I think using sample weights should help, and this answer doesn't seem to address that part of the OP (though what it does address is helpful)... $\endgroup$ – Ben Reiniger Jul 3 '19 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.