1
$\begingroup$

It seems to me that the mean IOU is a poor metric in the presence of unbalanced classes. E.g., suppose I have 10 classes but one image has only 2 classes present in its label. Consider the prediction where the 2 classes are inverted, the IOU for these classes is 0, but the IOU for the 8 other classes is 0/0. If we consider it to be 1 (since no prediction for those classes is the correct prediction), with uniform weights the resulting mean IOU is 0.8 which makes it sound like the prediction was quite good.

It seems that the mean IOU weighted by the proportion of pixels corresponding to each class in the target image solves this problem. Are there cases where using the later over uniform mean IOU is problematic? Why is the uniform mean IOU the more popular of the two metrics?

$\endgroup$
1
$\begingroup$

You somehow misunderstood how mean_IOU is calculated.

First, mean_IOU is not calculated per image and then averaging over all images, but calculated per class for images and then averaging over classes. That means for a single image, if some classes are missing (in both ground truth and your predicted labels), you don't need to worry about the 0/0 situation. You need to only update the confusion matrix, until you finished with all images, then calculate the mean_IOU.

In the extreme case, some classes are missing in all images. In that case, you should adjust you num_class parameter. Even if you don't, the missing classes will be treated as IOU=0 instead of 1 (sad...)

For example, suppose your image has 5 pixels and you have 3 classes. The ground truth vector is always [0,0,1,1,1] and you always predicted [1,1,0,0,0] (note that you predicted class 0 and 1 inverted, and class 2 is missing). In this case, the mean_IOU is 0, not 1/3.

Therefore, you should understand that the IOU is already being normalized by the sum of the numbers of pixies in (the union of your predicted pixels and the ground truth pixels) across all you images. So I don't think it's really necessary to re-weight the IOU any more.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.