I am training a Unet model for cell image segmentation from microscopy images. In order to help the model generalize better to different microscopes, I attempted to apply brightness augmentation to the training data, randomly multiplying images by a value chosen between (0.5, 1.5), then normalizing each image.
However, adding this augmentation always caused the model to get stuck at a very high loss (using jaccard loss). Interestingly, in the absence of this augmentation, the model converged to a jaccard score of 0.50. A similar effect was observed when applying random Gaussian blurrin: the model failed to learn. On the contrary, spatial augmentations such as flipping, rotating, etc did not have this effect.
I performed extensive testing on my code verified that the image augmentation function worked correctly and that the cells were still recognizable from blurred/darkened/brightened images.
I thought that the model cannot escape some local minima in the loss landscape. So, I experimented with a variety of weight initialization including HeNormal and HeUniform. I also tried first training first without augmentation and then using the weights to train with the brightness and blurring augmentations. Nothing worked. I am using Relu activation function and Adam optimizer. I tried to switch to SGD to see if it would help, but to no avail.
