I'm training Unet+MobileNetV3 for semantic segmentation objects on real photos using custom dataset and get strange results. I have already accumulated pretty big dataset and constantly improve it by adding new samples but the result still has mistakes that looks like not dataset issue. For numbers, I have over 20000 samples and additionaly use augmentations, aslo train for over 150 epochs. Loss and iou metrics looks fine. I'm sure that dataset is good, I get good results on real cases but get sudden bad results too and can't figure out why. Putting aside samples that completely missed, I get partially segmented objects and find it strange, because the objects has solid area but only part of it is found. Take a look at picture reference below (not actual data):

bad area

What I have already tried:

  • Deeper nets. I have tried different backbones for Unet but hadn't got any positive results regarding the stated problem.

  • Diferrent hyperparameters such as learning rate, epochs, schedules, batch sizes.

  • Increase dataset by adding wrong samples to it. Results are slightly better but the process seems endless and I found myself spotting new problems where everything was segmented fine several net versions before.

  • Reduce number of segmentation classes. I went down from 11 to 2 and still have the same problem.

  • Apply different augmentations to train dataset such as mirror, compression, random crop, color changes, brightness changes and etc.

  • Use pretrained weights and learn from scratch.

And now I'm stuck, because I can get 9 out of 10 real samples fine and 1 that look like other 9 but with bad segmentation results. It can be two photos of flat object taken at the same time but from different angle and camera (almost like simply mirrored), and only one of these segmentions is fine. What else I can research with my data and models to find out what is wrong? Maybe this is simply the problem of Unet or semantic segmentation itself?

I'm using repo segmentation_models.pytorch for nets and training.

  • $\begingroup$ Did you apply the same (geometric) augmentations to your ground truth mask too ? Otherwise it can completly mess up the training. $\endgroup$
    – Lelouch
    Feb 29 at 15:21
  • $\begingroup$ Yes, i'm checking everything I put into training or prediction because at first I had a lot of issues with it. Also I'm using repo albumentations thats has augmentations devided into image only and image+mask, this makes it difficult to miss mask augmentation when needed. Moreover, prediction in production is done in c++ and TensorRT, converting net model to onnx and then to TensorRT, and is equal to my python prediction. $\endgroup$ Mar 1 at 6:24


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.