1
$\begingroup$

I am working on training DNNs on satellite data. The class distribution in the data is extremely imbalanced, so I train the neural networks using random majority undersampling to artificially balance the number of training examples from each class. During validation, I don't resample the data in any way.

The loss as a function of epoch. Orange is training loss and blue is validation.

Accuracy over epoch. Orange is training, blue is validation.

In the graphs above, orange is training performance and blue is validation. The validation accuracy and loss values are much much noisier than the training accuracy and loss. Validation accuracy even hit 0.2% at one point even though the training accuracy was around 90%. Why are the validation metrics fluctuating like crazy while the training metrics stay fairly constant?

Some additional information on the model/data:

I'm segmenting satellite images into three land cover classes using a U-Net model. The U-Net model has half the number of filters at each layer compared to the original paper, and I apply batch normalization after each convolution and before the activation. The model is trained from scratch with the Adam optimizer with an initial lr=0.001 and a step decay schedule, optimizing (unweighted) categorical crossentropy. The U-Net has l2 weight regularization applied on each convolutional layer with constant 0.01.

The training and validation data are all floating point images with values in the range 0-2. The images for the two datasets were sampled from the same spatial region. I don't do any preprocessing or data augmentation because of the computational overhead.

The validation set has 317k pixels in class 1, 26.5M pixels in class 2, and 27.8M pixels in class 3 (this is the "true" data distribution). The number of training instances which contain pixels in class 1 is ~200 out of 3500. The training set (in which class 1 is spatially oversampled to artificially balance the training set), consists of 58M class 1 pixels, 166M class 2 pixels, and 127M class 3 pixels. Out of 20k training instances, approximately 8200 contain class 1 pixels.

When feeding data to the model, I sample from the three datasets with probability 0.4, 0.3, and 0.3 for classes 1, 2, and 3 respectively.

EDIT: Problem solved! When working with neural networks and geospatial data it's very important to maintain the same map projection between the test/train/validation sets. I thought the software I was using to extract the test/train/validation data was ensuring the same projection between sets, but I was wrong. Making the projections of the test/train/validation data the same fixed my problem. Essentially I was validating on a different dataset than I was training on, which is not how machine learning models are supposed to be validated.

$\endgroup$
2
  • $\begingroup$ Welcome to DS SE! Would you be able to share some details about your model and the data? Some particularly helpful things are: model architecture (and whether you are transfer learning), hyperparameters, number of instances per train/val, class distributions in train/val. As this post stands currently, there could be many possible answers, so these details can help people narrow down the problem. gl! $\endgroup$
    – Ben
    Sep 1, 2020 at 3:53
  • $\begingroup$ Thanks! I edited the question to better describe the experiment. $\endgroup$
    – Tom C
    Sep 1, 2020 at 14:01

1 Answer 1

1
$\begingroup$

There can be a few reasons for this behavior as already pointed out in "Why is the validation accuracy fluctuating?":

  • Size of train / validation stes: Fluctuations may become stronger the smaller your validation set is, especially during the early stage of training where predictions are closer to random predictions.

  • Overfitting: Train loss seems to be low and accuracy over 90%. If you're overfitting, the predictions on the validation set become almost random and you observe such fluctuations.

  • Random validation set: If you're validation set is not constant across all epochs, the validation loss will be different and losses from different epochs cannot be compared anymore. Especially if you have imbalanced classes.

  • Weighted loss function: Using a weighted loss function for imbalanced data sets makes your loss appear more stochastically, especially in the early stage of training and if the distributions of the training and validation sets mismatch.

Maybe you can share additional information to your data, preprocessing, training, model etc. so it is easier to spot the cause of this more precisely.

$\endgroup$
1
  • $\begingroup$ Thanks! I edited the question above. $\endgroup$
    – Tom C
    Sep 1, 2020 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.