Class distribution discrepancy training/validation. Loss now uninterpretable?

Question

I have a 3-class image classification problem. The classes are highly unbalanced (with about 98% of the images beloning to one class). To counteract this unbalanced data, I weight the losses by using the inverse of the class probabilities (1/(Proportion of class)). This means that the loss gets multiplied a great lot for the minority classes. This does help in counteracting the class imbalance.

However, for this weighted loss I use the proportions in the training dataset. In the validation dataset, the distribution is somewhat different (96.66%, 2.59%, 0.75% in val set as opposed to 98.28%, 0.98%, 0.73% in the training set). This means that if I use the weights based on the training set, mainly misclassified class 1 images gets punished a lot more than they should according to the distribution in the validation set.

The reason for this distribution discrepancy is that I can't just do a random split on my images, since they are taken by some 10 cameras all taking pictures of the same respective static scenery (highways). This means that I have to hold out all images of about 2/3 cameras as validation and use the other cameras for training, otherwise the model will be too optimistic, since validation images come from the same camera as the train images.

This makes the weighted loss uninterpretable for the validation data. Is there a way to counteract this? For instance, would giving separate weights for the train/validation sets (based on their respective distributions) help?

Answer 1

The validation set is used to estimate how well the model is fitting to your solution space. For instance, if your training loss continues to decrease but your validation loss remains constant or increases, then you should employ early stopping as to avoid overfitting. This does not require a weighted loss.

If you wanted to compare the training loss to the validation loss, you cannot use the same weights if the distributions differ. Instead, you can simply compare the average unweighted loss per set. The weighted loss is for penalization.