In order to familiarize myself with semantic segmentation and convolutional neural networks I am going through this tutorial by MathWorks:

Semantic Segmentation Using Deep Learning

I did not use the pretrained version of Segnet since I wanted to test on my custom data set. All code is the same, however I have different classes, and fewer labels. Below image shows the label name and amount of pixels associated with each.

enter image description here

To make up for the low pixel data for class 2, median frequency balancing was performed.

imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount
classWeights = median(imageFreq) ./ imageFreq

I proceed to train the network using the code provided in the example with the options and lgraph unchanged. The SegNet network is created with weights initialized from the VGG-16 network.

Unlike the example, I get a much lower global accuracy:

enter image description here

To gain further insight I plotted the Mini-batch accuracy and Mini-batch loss against each iteration.

enter image description here

It is clearly seen that the accuracy fluctuates wildly and ends up worse than it started, so the network learned absolutely nothing! However the loss decreased gradually.

A possible solution I propose would be to use inverse frequency balancing. However, in the example above, median frequency balancing was already performed, so I doubt how much this would help.

Is the terrible performance related to simply not having enough training data? Can anything be be done to improve performance with existing data?

Any suggestions are greatly appreciated.


2 Answers 2


If you provided a validation accuracy on the plot - the first idea is that it is overfitting. If that is true, you need to try two things first:

  • Add more augmentations (or increase the dataset). I do not know how to do this in Matlab, but you can see a list of common image augmentations here.
  • Early stopping

Possible tricks which can improve the accuracy of semantic segmentation models:

  • Try lowering the learning rate
  • Try variating the batch size
  • Train longer (more epochs)
  • Train the model on larger images
  • Increase the dataset size

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.