Multiclass semantic segmentation with some classes possibly not present in some of the images

Question

Let's assume we have a large annotated dataset with 4 classes. In this dataset, there might be annotated images with less than 4 classes, where the remaining classes might or might not be present. As an example, say we want to detect pedestrians/cars/bicycles/roads in images. In our dataset, there are some annotated images with only 3 classes: pedestrians/cars/bicycles, but this does not mean there are no roads in these images. That is, there might be roads in these images that were ignored by the annotator for some reason, or there might no be roads at all. My question is, how do we take this uncertainty into the loss term?

One option is to work with independent networks for each class. But what if we want to train a single network? how do we add something like a "don't care" term for objects not annotated in an image, but might still be present in the image?

Answer 1

That problem is commonly called weak labels and is associated with weak supervision.

Weak labels are often not handled by the loss function. Most conventional loss functions assume perfectly labeled data points and have no mechanism to handle label uncertainty.

An alternative framework for weak labels is Confident Learning which

characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence.