2
$\begingroup$

I wanted a metric where I could weigh each class as I wish while measuring "total accuracy". sklearn seems to have this with balanced_accuracy_score. Irrespective of the sample_weight, I am getting the same "balanced accuracy". Why? what was the point of sample_weights?

from sklearn.metrics import balanced_accuracy_score

sample_weight = np.array([1 if i == 0 else 1000 for i in y])
balanced_accuracy_score(y,m.predict(xs),sample_weight=sample_weight)

Here are the docs.

$\endgroup$

1 Answer 1

3
$\begingroup$

The point of sample_weights is to give weights to specific sample (e.g. by their importance or certainty); not to specific classes.
Apparently, the "balanced accuracy" is (from the user guide):

the macro-average of recall scores per class

So, since the score is averaged across classes - only the weights within class matters, not between classes... and your weights are the same within class, and change only across classes.

Explicitly (from the user guide again):

$$\hat{w}_i = \frac{w_i}{\sum_j{1(y_j = y_i) w_j}}$$

i.e. the i-th sample is re-weighted by dividing its weight by the total weights of samples with the same label.

Now, if you want, you can just use the simple accuracy score, and plug in weights as you see fit.

In the following example:

from sklearn.metrics import balanced_accuracy_score, accuracy_score

y_true = [0, 1, 0, 0, 1, 0, 1, 1, 1, 1]
y_pred = [0, 1, 0, 0, 0, 1, 1, 1, 1, 1]

some_sample_weights =[10, 1, 1, 1, 10, 1, 0.5, 0.5, 0.5, 0.5]
weights_by_class =[1 if y==1 else 1000 for y in y_true]

print('with some weights: {:.2f}'.format(balanced_accuracy_score(y_true, y_pred, sample_weight=some_sample_weights)))
print('without weights: {:.2f}'.format(balanced_accuracy_score(y_true, y_pred)))
print('with class weights in balanced accuracy score: {:.2f}'.format(balanced_accuracy_score(y_true, y_pred, sample_weight=weights_by_class)))
print('with class weights in accuracy score: {:.5f}'.format(accuracy_score(y_true, y_pred, sample_weight=weights_by_class)))

class_sizes = [sum((1 for y in y_true if y==x))/len(y_true) for x in (0,1)] 
weights_by_class_manually_balanced = [w/class_sizes[y] for w, y in zip(weights_by_class, y_true)]

print('with class weights in accuracy score (manually balanced): {:.5f}'.format(accuracy_score(y_true, y_pred, sample_weight=weights_by_class_manually_balanced)))

you get:

with some weights: 0.58
without weights: 0.79
with class weights in balanced accuracy score: 0.79
with class weights in accuracy score: 0.75012
with class weights in accuracy score (manually balanced): 0.75008

As you can see:

  • using class weights in the balanced accuracy score didn't matter; they just got adjusted back into class sizes.
  • using class weights in the accuracy score is very close to 75% (3 of out of 4 the 0 labels are correctly classified), and re-adjusting the weights according to class sizes doesn't matter much (the accuracy is a bit less, because the 0 class is larger)
$\endgroup$
3
  • $\begingroup$ "It is the macro-average of recall scores per class or, equivalently, raw accuracy where each sample is weighted according to the inverse prevalence of its true class.". I don't get it, why you are saying then that weights within the class only are important. I can see it in your example, but I don't get why the "weights doesn't work" across classes. I was expecting (as this is for unbalanced datasets) that it would do a weighted average. And the only way to give input is by using sample_weights. $\endgroup$
    – Pandian Le
    Jan 3, 2021 at 10:17
  • $\begingroup$ I added the explicit calculation (from the user guide) that shows explicitly why the weights don't work across classes. I think you might want to derive your own score (do the macro-average of recall scores as a weighted average, not average by class sizes); the balanced-accuracy-score isn't what you need. Maybe just take the accuracy score and divide your weights by the class weights? $\endgroup$ Jan 5, 2021 at 7:44
  • $\begingroup$ I've edited my answer to explain how to use custom weights in accuracy score. $\endgroup$ Jan 5, 2021 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.