# Unbalanced multiclass data with XGBoost

I have 3 classes with this distribution:

Class 0: 0.1169
Class 1: 0.7668
Class 2: 0.1163


And I am using xgboost for classification. I know that there is a parameter called scale_pos_weight.

But how is it handled for 'multiclass' case, and how can I properly set it?

scale_pos_weight is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a value to scale_pos_weight is:

sum(negative instances) / sum(positive instances)


For your specific case, there is another option in order to weight individual data points and take their weights into account while working with the booster, and let the optimization happen regarding their weights so that each point is represented equally. You just need to simply use:

xgboost.DMatrix(..., weight = *weight array for individual weights*)


You can define the weights as you like and by doing so, you can even handle imbalances within classes as well as imbalances across different classes.

• > A good approach when assigning a value to scale_pos_weight is: sum(negative instances) / sum(positive instances) Oct 8, 2019 at 14:09
• I see this advice everywhere and it make sense to assign a higher weight to the less represented class. However I have a hard time finding a source discussing this exact value. I get the intuition behind that specific value (make the sample balanced) but I suspect there is a variance trade-off somewhere, that would make you want consider lower weight. Oct 8, 2019 at 14:27
• Personally @Kerem it is still unclear to me if the positive instances are the majority or the minority classes. Do we assume that always positive=majority and negative=minority and hence sum(negative instances)/sum(positive instances)=sum(minority)/sum(majority)? Feb 21, 2020 at 12:36
• This is attempt to clarify the doubt raised by @Outcast. Effectively scale_pos_weight scales the error of positive cases. if the positive cases are majority in then value of scale_pos_weight < 1 and hence positive case errors will be scaled down. However if positive cases are minority then scale_pos_weight > 1 and hence positive case errors will be scaled up. Hope it clarifies. Oct 14, 2020 at 9:13
• @Kerem T How can you use it inside a pipeline? Nov 11, 2021 at 8:35

This answer by @KeremT is correct. I provide an example for those who still have problems with the exact implementation.

weight parameter in XGBoost is per instance not per class. Therefore, we need to assign the weight of each class to its instances, which is the same thing.

For example, if we have three imbalanced classes with ratios

class A = 10%
class B = 30%
class C = 60%


Their weights would be (dividing the smallest class by others)

class A = 1.000
class B = 0.333
class C = 0.167


Then, if training data is

index   class
0       A
1       A
2       B
3       C
4       B


we build the weight vector as follows:

index   class    weight
0       A        1.000
1       A        1.000
2       B        0.333
3       C        0.167
4       B        0.333

• thanks for this, how does this affect the probability output for xgboost? i.e. when we come to optimize the loss function does the weight act on the optimizartion of the loss? Sep 2, 2020 at 13:40
• This example is a bit misleading, if you feed this data now as input to your classifier then you will get a 100% accuracy since you have already told the model what the actual classes are Nov 29, 2020 at 12:07
• @Guarav, not at all correct. The weights are not features for the model. They influence the gradients or error function only. Feb 10, 2021 at 21:32

For sklearn version < 0.19

Just assign each entry of your train data its class weight. First get the class weights with class_weight.compute_class_weight of sklearn then assign each row of the train data its appropriate weight.

I assume here that the train data has the column class containing the class number. I assumed also that there are nb_classes that are from 1 to nb_classes.

from sklearn.utils import class_weight
classes_weights = list(class_weight.compute_class_weight('balanced',
np.unique(train_df['class']),
train_df['class']))

weights = np.ones(y_train.shape[0], dtype = 'float')
for i, val in enumerate(y_train):
weights[i] = classes_weights[val-1]

xgb_classifier.fit(X, y, sample_weight=weights)


Update for sklearn version >= 0.19

There is simpler solution

from sklearn.utils import class_weight
classes_weights = class_weight.compute_sample_weight(
class_weight='balanced',
y=train_df['class']
)

xgb_classifier.fit(X, y, sample_weight=classes_weights)

• You can use now simply compute_sample_weight('balanced', y) from sklearn.utils Feb 2, 2021 at 20:42
• What about the use inside a Pipeline? Nov 11, 2021 at 8:35

Everyone stumbles upon this question when dealing with unbalanced multiclass classification problem using XGBoost in R. I did too!

I was looking for an example to better understand how to apply it. Invested almost an hour to find the link mentioned below. For all those who are looking for an example, here goes.

Thanks wacax

• Can you add some description of what the link is? Otherwise this is a link-only answer. Your bottom line is xgb.DMatrix(..., weight) has an instance-specific (not per-class) weight argument, and you can tweak it per-instance.
– smci
Mar 13, 2020 at 2:56

The parameter scale_pos_weight works for two classes (binary classification).

The parameter weight goes into the xgb.DMatrix function can be used for three or more classes. The weights can be computed like this:

weights = total_samples / (n_classes * class_samples * 1.0)