I have a 3-class image classification problem. The classes are highly unbalanced (with about 98% of the images beloning to one class). To counteract this unbalanced data, I weight the losses by using the inverse of the class probabilities (1/(Proportion of class)). This means that the loss gets multiplied a great lot for the minority classes. This does help in counteracting the class imbalance.
However, for this weighted loss I use the proportions in the training dataset. In the validation dataset, the distribution is somewhat different (96.66%, 2.59%, 0.75% in val set as opposed to 98.28%, 0.98%, 0.73% in the training set). This means that if I use the weights based on the training set, mainly misclassified class 1 images gets punished a lot more than they should according to the distribution in the validation set.
The reason for this distribution discrepancy is that I can't just do a random split on my images, since they are taken by some 10 cameras all taking pictures of the same respective static scenery (highways). This means that I have to hold out all images of about 2/3 cameras as validation and use the other cameras for training, otherwise the model will be too optimistic, since validation images come from the same camera as the train images.
This makes the weighted loss uninterpretable for the validation data. Is there a way to counteract this? For instance, would giving separate weights for the train/validation sets (based on their respective distributions) help?