Extremely stochastic validation loss/accuracy

Question

I am working on training DNNs on satellite data. The class distribution in the data is extremely imbalanced, so I train the neural networks using random majority undersampling to artificially balance the number of training examples from each class. During validation, I don't resample the data in any way.

In the graphs above, orange is training performance and blue is validation. The validation accuracy and loss values are much much noisier than the training accuracy and loss. Validation accuracy even hit 0.2% at one point even though the training accuracy was around 90%. Why are the validation metrics fluctuating like crazy while the training metrics stay fairly constant?

Some additional information on the model/data:

I'm segmenting satellite images into three land cover classes using a U-Net model. The U-Net model has half the number of filters at each layer compared to the original paper, and I apply batch normalization after each convolution and before the activation. The model is trained from scratch with the Adam optimizer with an initial lr=0.001 and a step decay schedule, optimizing (unweighted) categorical crossentropy. The U-Net has l2 weight regularization applied on each convolutional layer with constant 0.01.

The training and validation data are all floating point images with values in the range 0-2. The images for the two datasets were sampled from the same spatial region. I don't do any preprocessing or data augmentation because of the computational overhead.

The validation set has 317k pixels in class 1, 26.5M pixels in class 2, and 27.8M pixels in class 3 (this is the "true" data distribution). The number of training instances which contain pixels in class 1 is ~200 out of 3500. The training set (in which class 1 is spatially oversampled to artificially balance the training set), consists of 58M class 1 pixels, 166M class 2 pixels, and 127M class 3 pixels. Out of 20k training instances, approximately 8200 contain class 1 pixels.

When feeding data to the model, I sample from the three datasets with probability 0.4, 0.3, and 0.3 for classes 1, 2, and 3 respectively.

EDIT: Problem solved! When working with neural networks and geospatial data it's very important to maintain the same map projection between the test/train/validation sets. I thought the software I was using to extract the test/train/validation data was ensuring the same projection between sets, but I was wrong. Making the projections of the test/train/validation data the same fixed my problem. Essentially I was validating on a different dataset than I was training on, which is not how machine learning models are supposed to be validated.

Answer 1

There can be a few reasons for this behavior as already pointed out in "Why is the validation accuracy fluctuating?":

• Size of train / validation stes: Fluctuations may become stronger the smaller your validation set is, especially during the early stage of training where predictions are closer to random predictions.

• Overfitting: Train loss seems to be low and accuracy over 90%. If you're overfitting, the predictions on the validation set become almost random and you observe such fluctuations.

• Random validation set: If you're validation set is not constant across all epochs, the validation loss will be different and losses from different epochs cannot be compared anymore. Especially if you have imbalanced classes.

• Weighted loss function: Using a weighted loss function for imbalanced data sets makes your loss appear more stochastically, especially in the early stage of training and if the distributions of the training and validation sets mismatch.

Maybe you can share additional information to your data, preprocessing, training, model etc. so it is easier to spot the cause of this more precisely.