1
$\begingroup$

I'm working with an unbalanced classification problem, in which the target variable contains:

np.bincount(y_train)
array([151953,  13273])

i.e. 151953 zeroes and 13273 ones.

To deal with this I'm using XGBoost's weight parameter when defining the DMatrix:

dtrain = xgb.DMatrix(data=x_train, 
                     label=y_train,
                     weight=weights)

For the weights I've been using:

bc = np.bincount(y_train)
n_samples = bc.sum()
n_classes = len(bc)
weights = n_samples / (n_classes * bc)
w = weights[y_train.values]

Where weightsis array([0.54367469, 6.22413923]), and with the last line of code I'm just indexing it using the binary values in y_train. This seems like the correct approach to define the weights, since it represents the actual ratio between the amount of values of one class vs the other. However this seems to be favoring the minoritary class, which can be seen by inspecting the confusion matrix:

array([[18881, 19195],
       [  657,  2574]])

So just by trying out different weight values, I've realized that with a fairly close weight ratio, specifically array([1, 7]), the results seem much more reasonable:

array([[23020, 15056],
       [  837,  2394]])

So my question is:

  • Why using the actual weights of each class yielding poor metrics?
  • Which is the right way to set the weights for an unbalanced problem?
$\endgroup$

2 Answers 2

2
$\begingroup$

Depending on your choice of accuracy metric, you'll find that different balancing ratios give the optimum value of the metric. To see why this is true, consider optimizing precision alone vs. optimizing recall alone. Precision is optimized (=1.0) when there are no false positives. Upweighting negative data reduces the positive rate, and therefore the false positive weight. So if you just want to optimize precision, give the positive data zero weight! You'll always predict negative labels and the precision will be ideal. Likewise, for only optimizing recall, give the negative data zero weight - you'll always get the ideal value of recall. These extreme cases are silly for real-world applications, but they do show that your "best" balancing ratio depends on your metric.

As you're probably aware, metrics like AUC and F1 try to compromise between precision and recall. In the absence of prior information, people often try to choose "equal balance" between precision and recall, as implemented in AUC. Since AUC is relatively insensitive to data balance, 1:1 data balancing is generally appropriate. However, in real life you may care more about precision than recall, or vice versa. So, you do need to select your metric in advance, depending on the problem you're solving. Then keep your metric fixed, vary your data balance, and look at your trained model performance on realistic test datasets. Then you can see whether your model is making the optimum predictions, from the point of view of your chosen metric and your real-world dataset.

$\endgroup$
3
  • $\begingroup$ Thanks a lot for your answer. By varying the balance you mean I could vary the weights having fixed a metric, for instance AUC, and check the results to keep an optimal value? $\endgroup$
    – yatu
    Mar 11, 2020 at 10:50
  • $\begingroup$ yes - keep in mind that you are optimizing the weights on your training set so that you get the best result on your unweighted test set. $\endgroup$ Mar 11, 2020 at 18:02
  • $\begingroup$ Thanks a lot Dave :) $\endgroup$
    – yatu
    Mar 12, 2020 at 10:24
0
$\begingroup$

Instance Weight File

XGBoost supports providing each instance an weight to differentiate the importance of instances. For example, if we provide an instance weight file for the "train.txt" file in the example as below:

train.txt.weight

1

0.5

0.5

1

0.5

It means that XGBoost will emphasize more on the first and fourth instance, that is to say positive instances while training. The configuration is similar to configuring the group information. If the instance file name is "xxx", XGBoost will check whether there is a file named "xxx.weight" in the same directory and if there is, will use the weights while training models.

  1. Important does not always equate with balanced.
  2. Dont even set weights, just make sure the problem is balanced, there are a ton of recources on this.
$\endgroup$
5
  • $\begingroup$ Dont even set weights, just make sure the problem is balanced. You're suggesting then that finding a proper sample weight will never be better than over/under sampling techniques to balance the problem..? $\endgroup$
    – yatu
    Mar 10, 2020 at 15:12
  • $\begingroup$ This is not a binary answer. "never" never works with ML. There are datasets where linear regression is better than NN. Time and cost of trying to optimise these parameters directly is not worth it. Sure you could find optimum, and obviously not so trivially as you tried. $\endgroup$
    – Noah Weber
    Mar 10, 2020 at 15:14
  • $\begingroup$ Well trivial or not I'm using the recommended approach in XGboost's docs when specifying a weight parameter. So I'd be expecting a fairly better "balanced result" by applying these weights than the one I've obtained. $\endgroup$
    – yatu
    Mar 10, 2020 at 15:18
  • 1
    $\begingroup$ In any case I'm expecting some guidance/help on how to better set these weights, rather than to tackle the problem in a whole different way $\endgroup$
    – yatu
    Mar 10, 2020 at 15:19
  • $\begingroup$ i've frequently observed that 1:1 balance does not give the best result, see my answer below for how to set balance optimally for your model. $\endgroup$ Mar 10, 2020 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.