Batch Normalization and Dropout together causing incorrect segmentation results

Question

So, I've been running a test to see how well a number of networks can perform road segmentation on a particular customer's dataset. I am testing UNET, RDRCNN, and Tiramisu against each other.

UNET reliably trains, however RDRCNN and Tiramisu cannot. Immediately during training, the validation loss and accuracy plateau. After looking at the results, it looks as if the network is guessing a single pixel value; leading to inferences that are just a square of all one color.

I have tried turning the learning rate down, using a simple optimizer like SGD, and making sure I am loading the data with the same exact loader. I have also tried using a dice/jaccard loss to account for class imbalance. None of these have worked.

Has anyone ever experienced this before? Any help would be greatly appreciated. Thank you!

Answer 1

The differences between UNET and the deeper networks like RDRCNN and Tiramisu is that the deeper networks have Batch Normalization layers. They also include dropout.

I thought it might be the bias terms in the convolutional layers, but the bias for all those layers was set to 'False'. I tried regularization, changing the momentum value of the Batch Norm layers, and removing the Batch Norm layers closest to the output. However, the only thing that worked was removing all the Batch Norm layers entirely or removing all the dropout entirely.

It seems this is a data issue on my end. Certain images are exacerbating a common problem with using batch normalization and dropout together. For some, this may introduce too much noise into the network, leading to incorrect predictions.