Semantic segmentation of an image with multiple labels per pixel

Question

I am building a model for a multiclass sematic segmentation of a skin disease. At a moment I am using U-Net for binary classifications.

In this multiclass problem I have the following cases. There are four types of skin damage. There are four degrees of damage for each skin damage type: healthy, mild, moderate, severe. Healthy skin is A0_B0_C0_D0, "mild b and severe c" corresponds to A0_B1_C3_D0. I would like to train a single multiclass model to predict a dictionary {A: scoreA, B: scoreB, C: scoreC, D: scoreD}. Given that

4(damage types)**4(damage degrees) = 256 combinations

Do I need to train a 256 class model? My concern is that I only have 200 images in my training set and some of the combinations will not appear at all in the training set. Is there a way to train a 12 class model returning four values corresponding to the most likely damage type for each damage type?

Update. Consider an rgb image. You want to classify brightness of each color channel in categories

Intensity   class
 0-63 -> "0"
64-127 -> "1"
128-191 -> "2" 
192-255 -> "3" 

Then each pixel belongs to one of 64 classes (r0g0b0, r0g0b1, ..., r3g2b2, r3g3b3). The training set has pixels of colors r0, r1, r2, r3, g0, g1, g2, g3, b0, b1, b2, b3, but it has no pixels of color r0g1b2 or of color r2g3b0. Three separate models (one per channel) will easily learn to predict the channel category, but it will never output r0g1b2 and r2g3b0 classes in 64 class model because it have never seen those classes. How to overcome this problem?

Answer 1

Three separate models (one per channel) will easily learn to predict the >channel category, but it will never output r0g1b2 and r2g3b0 classes in 64 class model because it have never seen these classes. How to overcome this problem?

The only way to solve this problem is to use channels (one for each skin damage type) for each pixel, and treat it as a regression rather than a classification.

In other words, use a multi output regression.

For your output, use a convolutional layer that gives the same number of rows and columns as the input, but 4 channels (one for each skin damage type).

Your ground truth (y_true) should be an array the same width and height as your input, but with 4 channels (one for each skin damage type) with each holding the severity rating of that pixel for the corresponding skin damage type.

Your loss function could be something used for regression such as MAE (mean absolute error).

This is because a classification will output 0 for classes it has never seen samples of since that is what minimizes the loss. A regression, on the other hand, will treat the target variable as continuous, and even if it hasn't seen examples of all severity levels of Type A skin damage (for example), it can still output them.

You could then choose thresholds to classify the outputs into the corresponding labels.

Ex. for each output <0.5 is 0, >0.5 is 1, >1.5 is 2, >2.5 is 3, >3.5 is 4

so [0.1, 1.6, 0.7, 5] means {A: 0, B: 2, C: 1, D: 4}

This is also more useful and easier to interpret in a clinical context as the severity is ordinal rather than just categorical, so a doctor would be better off knowing that a particular mild case (for example) had a prediction of 1.4 rather than 0.6, because they both correspond to a mild prediction but a 1.4 is closer to moderate and may be treated differently than a 0.6

This is an interesting problem from a learning and research perspective, but I develop deep learning based prognosis and diagnosis models using medical images at a big pharma/life sciences company and can tell you that a dataset of 200 images for a task this complex will be insufficient for decent performance or reliable results. multi-fold training/validation/testing AND some slight image augmentation will be necessary, but probably still insufficient.