I've implemented a SegNet and SegNet ReLU variant in PyTorch. I'm using it as a proof-of-concept for now, but what really bothers me is the noise produced by the network. With ADAM I seem to get slightly less noise, whereas with SGD the noise increases. I can see the loss going down and the cross-evaluation accuracy rising to 98%-99% and yet the noise is still there.

On the left is the actual image, then you can see the mask, and finally the actual output from the network. There's 1024 samples per class, and two classes, which are very consistent as the documents are very structured. I'm using the vanilla SegNet (same kernel, striding and padding) on 224x224.

What could explain this noise, and how could I potentially address the issue?

InputMasked OutputActual output

  • $\begingroup$ Are you following some readymade code for this approach? Can you share that? Previously I have done similar using opencv, by eroding and thresholding to get the text block boundaries (non ML approach). $\endgroup$ Commented Mar 8, 2020 at 7:04
  • $\begingroup$ @SandeepBhutani I can't share my code, but the original code is from github.com/say4n/pytorch-segnet/tree/master/src What seemed to work best, is using github.com/qubvel/segmentation_models.pytorch which is very nicely organised and supports multiple encoders. I got my best results with that. $\endgroup$
    – Ælex
    Commented Mar 9, 2020 at 9:53

1 Answer 1


I'm going to try and attempt to answer my question, but won't accept it as an answer, simply because I'm sure there is more than one reasons as to why this is happening. I've solved the issue by increasing the areas by adding more "features", e.g., I've made sure there is more text, table boxes and other visual features for the convolutions to "pick up". To a certain degree this seems to have helped a lot. What also helped, is using a more modern model, I've tried Unet with a resnet34 encoder and I also tried a DeepLabV3 which outperformed all others. So I suspect that the "noise" (for lack of a better word) is a by-product of the network being uncertain where exactly the boundaries of the segmentation are, due to absence of features. I suspect that the more modern models are better suited to deal with that problem.


So far what seems to have worked for me is DeepLabV3 with Resnet encoders, but another thing that really seems to make a difference is how "large" the segmented area is. Obviously in some domains this is not possible to adjust (e.g., robotics, autonomous vehicles and suchs) but in document or text processing it most likely is. What I've noticed is that after the downsizing to 224x224 smaller and thinner areas become very hard if not impossible to learn, whereas larger and thicker areas are easier. I suspect that averaging CELoss might be a culprit here, and that there may be a loss mechanism which emphasises errors in smaller regions/areas where errors still exist. The hypothesis is that the smaller regions will tolerate errors due to the averaging operation whereas larger ones won't tolerate the errors as much.

  • 1
    $\begingroup$ Generating exact visual segmentation is a very hard problem (not solved so far unless you have like unlimited amount of training data and computing power available), usually a bounding box segmentation is used. I guess the noise that you see, is because your model tries to get exact segmentation for which it has by far not enough training data. $\endgroup$
    – Eugen
    Commented Mar 5, 2020 at 8:25
  • $\begingroup$ Yes I think it tries to match the exact box. However I do have a lot of data, what I don't have is enough variation in that data which I suspect is the culprit here. However using a Deep Lab approach has helped tremendously. But the domain of textual processing is also a lot simpler than say autonomous driving. $\endgroup$
    – Ælex
    Commented Mar 5, 2020 at 15:16
  • $\begingroup$ You're right, just having unlimited or a lot of training data is a bad formulation from me, I guess more precise is the requirement of a huge amount of very high dimensional data and a lot of computing power to be able to apply more exact segmentations. Just noticed it on googlemaps while zooming in and watching the segmentation getting more exact on the roads and rivers. This is where the computing power and data of all our distributed android systems approaches the requirements of the dimensionality and the amounts, I guess. $\endgroup$
    – Eugen
    Commented Mar 6, 2020 at 7:39
  • 1
    $\begingroup$ I read about iterated function systems for fractal image compression, seems to be a very rewarding topic to focus on getting deeper into it. $\endgroup$
    – Eugen
    Commented Mar 6, 2020 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.