26
$\begingroup$

I have 3 classes with this distribution:

Class 0: 0.1169
Class 1: 0.7668
Class 2: 0.1163

And I am using xgboost for classification. I know that there is a parameter called scale_pos_weight.

But how is it handled for 'multiclass' case, and how can I properly set it?

$\endgroup$
22
$\begingroup$

scale_pos_weight is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a value to scale_pos_weight is:

sum(negative instances) / sum(positive instances)

For your specific case, there is another option in order to weight individual data points and take their weights into account while working with the booster, and let the optimization happen regarding their weights so that each point is represented equally. You just need to simply use:

xgboost.DMatrix(..., weight = *weight array for individual weights*)

You can define the weights as you like and by doing so, you can even handle imbalances within classes as well as imbalances across different classes.

| improve this answer | |
$\endgroup$
  • $\begingroup$ > A good approach when assigning a value to scale_pos_weight is: sum(negative instances) / sum(positive instances) $\endgroup$ – lcrmorin Oct 8 '19 at 14:09
  • 1
    $\begingroup$ I see this advice everywhere and it make sense to assign a higher weight to the less represented class. However I have a hard time finding a source discussing this exact value. I get the intuition behind that specific value (make the sample balanced) but I suspect there is a variance trade-off somewhere, that would make you want consider lower weight. $\endgroup$ – lcrmorin Oct 8 '19 at 14:27
  • $\begingroup$ Personally @Kerem it is still unclear to me if the positive instances are the majority or the minority classes. Do we assume that always positive=majority and negative=minority and hence sum(negative instances)/sum(positive instances)=sum(minority)/sum(majority)? $\endgroup$ – Outcast Feb 21 at 12:36
  • $\begingroup$ This is attempt to clarify the doubt raised by @Outcast. Effectively scale_pos_weight scales the error of positive cases. if the positive cases are majority in then value of scale_pos_weight < 1 and hence positive case errors will be scaled down. However if positive cases are minority then scale_pos_weight > 1 and hence positive case errors will be scaled up. Hope it clarifies. $\endgroup$ – Saket Oct 14 at 9:13
13
$\begingroup$

This answer by @KeremT is correct. I provide an example for those who still have problems with the exact implementation.

weight parameter in XGBoost is per instance not per class. Therefore, we need to assign the weight of each class to its instances, which is the same thing.

For example, if we have three imbalanced classes with ratios

class A = 10%
class B = 30%
class C = 60%

Their weights would be (dividing the smallest class by others)

class A = 1.000
class B = 0.333
class C = 0.167

Then, if training data is

index   class
0       A
1       A
2       B
3       C
4       B

we build the weight vector as follows:

index   class    weight
0       A        1.000
1       A        1.000
2       B        0.333
3       C        0.167
4       B        0.333
| improve this answer | |
$\endgroup$
  • $\begingroup$ thanks for this, how does this affect the probability output for xgboost? i.e. when we come to optimize the loss function does the weight act on the optimizartion of the loss? $\endgroup$ – Maths12 Sep 2 at 13:40
10
$\begingroup$

Just assign each entry of your train data its class weight. First get the class weights with class_weight.compute_class_weight of sklearn then assign each row of the train data its appropriate weight.

I assume here that the train data have the column class containing the class number. I assumed also that there are nb_classes that are from 1 to nb_classes.

from sklearn.utils import class_weight
class_weights = list(class_weight.compute_class_weight('balanced',
                                             np.unique(train['class']),
                                             train['class']))

w_array = np.ones(y_train.shape[0], dtype = 'float')
for i, val in enumerate(y_train):
    w_array[i] = class_weights[val-1]

xgb_classifier.fit(X, y, sample_weight=w_array)
| improve this answer | |
$\endgroup$
5
$\begingroup$

Everyone stumbles upon this question when dealing with unbalanced multiclass classification problem using XGBoost in R. I did too!

I was looking for an example to better understand how to apply it. Invested almost an hour to find the link mentioned below. For all those who are looking for an example, here goes.

Thanks wacax

| improve this answer | |
$\endgroup$
  • $\begingroup$ Can you add some description of what the link is? Otherwise this is a link-only answer. Your bottom line is xgb.DMatrix(..., weight) has an instance-specific (not per-class) weight argument, and you can tweak it per-instance. $\endgroup$ – smci Mar 13 at 2:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.