# Semantic segmentation of an image with multiple labels per pixel

I am building a model for a multiclass sematic segmentation of a skin disease. At a moment I am using U-Net for binary classifications.

In this multiclass problem I have the following cases. There are four types of skin damage. There are four degrees of damage for each skin damage type: healthy, mild, moderate, severe. Healthy skin is A0_B0_C0_D0, "mild b and severe c" corresponds to A0_B1_C3_D0. I would like to train a single multiclass model to predict a dictionary {A: scoreA, B: scoreB, C: scoreC, D: scoreD}. Given that

4(damage types)**4(damage degrees) = 256 combinations

Do I need to train a 256 class model? My concern is that I only have 200 images in my training set and some of the combinations will not appear at all in the training set. Is there a way to train a 12 class model returning four values corresponding to the most likely damage type for each damage type?

Update. Consider an rgb image. You want to classify brightness of each color channel in categories

Intensity   class
0-63 -> "0"
64-127 -> "1"
128-191 -> "2"
192-255 -> "3"


Then each pixel belongs to one of 64 classes (r0g0b0, r0g0b1, ..., r3g2b2, r3g3b3). The training set has pixels of colors r0, r1, r2, r3, g0, g1, g2, g3, b0, b1, b2, b3, but it has no pixels of color r0g1b2 or of color r2g3b0. Three separate models (one per channel) will easily learn to predict the channel category, but it will never output r0g1b2 and r2g3b0 classes in 64 class model because it have never seen those classes. How to overcome this problem?

Three separate models (one per channel) will easily learn to predict the >channel category, but it will never output r0g1b2 and r2g3b0 classes in 64 class model because it have never seen these classes. How to overcome this problem?

The only way to solve this problem is to use channels (one for each skin damage type) for each pixel, and treat it as a regression rather than a classification.

In other words, use a multi output regression.

For your output, use a convolutional layer that gives the same number of rows and columns as the input, but 4 channels (one for each skin damage type).

Your ground truth (y_true) should be an array the same width and height as your input, but with 4 channels (one for each skin damage type) with each holding the severity rating of that pixel for the corresponding skin damage type.

Your loss function could be something used for regression such as MAE (mean absolute error).

This is because a classification will output 0 for classes it has never seen samples of since that is what minimizes the loss. A regression, on the other hand, will treat the target variable as continuous, and even if it hasn't seen examples of all severity levels of Type A skin damage (for example), it can still output them.

You could then choose thresholds to classify the outputs into the corresponding labels.

Ex. for each output <0.5 is 0, >0.5 is 1, >1.5 is 2, >2.5 is 3, >3.5 is 4

so [0.1, 1.6, 0.7, 5] means {A: 0, B: 2, C: 1, D: 4}

This is also more useful and easier to interpret in a clinical context as the severity is ordinal rather than just categorical, so a doctor would be better off knowing that a particular mild case (for example) had a prediction of 1.4 rather than 0.6, because they both correspond to a mild prediction but a 1.4 is closer to moderate and may be treated differently than a 0.6

This is an interesting problem from a learning and research perspective, but I develop deep learning based prognosis and diagnosis models using medical images at a big pharma/life sciences company and can tell you that a dataset of 200 images for a task this complex will be insufficient for decent performance or reliable results. multi-fold training/validation/testing AND some slight image augmentation will be necessary, but probably still insufficient.

• What sort of input and output should it be? If I have 256x256 rgb images, do I pass Nx256x256x3 images along with NX256x256x4 masks with mask values ranging from 0 to 3? Feb 5 at 12:34
• Thank you for your response. What is the typical number of images sufficient for this task? Hundreds, thousands, millions? Feb 5 at 12:53
• hi @sixtytrees, I've edited my response to account for the fact that it's a segmentation problem. Regarding the number of images sufficient for this task I would say at least a few thousand, but then you would need to try and fine tune a pre-trained network since you would need more if you were to train a network from scratch. Can I ask what this project is for?
– Fab
Feb 5 at 16:16
• @sixtytrees I made another edit mentioning (at the end) that you should also use some slight image augmentation - this will help you prevent overfitting which is very likely when you use a complex model without enough samples. Ex. use keras imagedatagenerator with rotation_range = 10, zoom, height shift, width shift, and shear of 0.1, and horizotal flip. This will provide some realistic variations your model could expect to see in clinical images and hence reduce overfitting and improve test performance.
– Fab
Feb 5 at 16:31