Semantic segmentation: mean IOU in presence of missing classes - Data Science Stack Exchange most recent 30 from datascience.stackexchange.com 2022-01-27T11:08:06Z https://datascience.stackexchange.com/feeds/question/38764 https://creativecommons.org/licenses/by-sa/4.0/rdf https://datascience.stackexchange.com/q/38764 1 Semantic segmentation: mean IOU in presence of missing classes Le Frite https://datascience.stackexchange.com/users/59615 2018-09-25T12:21:21Z 2018-10-25T20:00:31Z <p>It seems to me that the mean IOU is a poor metric in the presence of unbalanced classes. E.g., suppose I have 10 classes but one image has only 2 classes present in its label. Consider the prediction where the 2 classes are inverted, the IOU for these classes is 0, but the IOU for the 8 other classes is 0/0. If we consider it to be 1 (since no prediction for those classes is the correct prediction), with uniform weights the resulting mean IOU is 0.8 which makes it sound like the prediction was quite good. </p> <p>It seems that the mean IOU weighted by the proportion of pixels corresponding to each class in the target image solves this problem. Are there cases where using the later over uniform mean IOU is problematic? Why is the uniform mean IOU the more popular of the two metrics?</p> https://datascience.stackexchange.com/questions/38764/-/38784#38784 1 Answer by user12075 for Semantic segmentation: mean IOU in presence of missing classes user12075 https://datascience.stackexchange.com/users/14006 2018-09-25T17:45:55Z 2018-09-25T17:45:55Z <p>You somehow misunderstood how mean_IOU is calculated. </p> <p>First, mean_IOU is not calculated <em>per image</em> and then averaging over all images, but calculated <em>per class</em> for images and then averaging over classes. That means for a single image, if some classes are missing (in both ground truth and your predicted labels), you don't need to worry about the 0/0 situation. You need to only update the confusion matrix, until you finished with all images, then calculate the mean_IOU.</p> <p>In the extreme case, some classes are missing in all images. In that case, you should adjust you num_class parameter. Even if you don't, the missing classes will be treated as IOU=0 instead of 1 (sad...)</p> <p>For example, suppose your image has 5 pixels and you have 3 classes. The ground truth vector is always [0,0,1,1,1] and you always predicted [1,1,0,0,0] (note that you predicted class 0 and 1 inverted, and class 2 is missing). In this case, the mean_IOU is 0, not 1/3.</p> <p>Therefore, you should understand that the IOU is already being normalized by the sum of the numbers of pixies in (the union of your predicted pixels and the ground truth pixels) across all you images. So I don't think it's really necessary to re-weight the IOU any more.</p>