Torchmetrics Binary Accuracy and Multiclass Accuracy don't match

Question

in my program I have the problem that for a 2-class classification problem my multiclass accuracy and binary class accuracy don't match. I have generated a very small sample example where you can see my parameter settings and outcomes:

from torchmetrics.classification import Accuracy
target = tensor([[0, 1], [1, 0], [0, 1]])
preds = tensor([[0.59, 0.91], [0.91, 0.99],  [0.63, 0.04]])

bin_acc = Accuracy(task="binary")
multi_acc = Accuracy(task="multiclass", num_classes=2)

print(bin_acc(preds, target))
print(multi_acc(preds, target))

The output is:

tensor(0.3333)
tensor(0.)

The binary accuracy value is the one that I would expect but value from multiclass accuracy is wrong. Can anyone tell what I am doing wrong?

Answer 1

Looking at the source code for MulticlassAccuracy, it seems that the target should be a tensor of shape [N] with the target class label in each coordinate, instead of probabilities. In your case:

target_class_labels = tensor([1, 0, 1])
print(multi_acc(preds, target_class_labels))

will print the desired result.