Class Imbalance in Dataset of Images

Question

When dealing with an imbalanced dataset, I have been taught to oversample on only the train samples and not the entire dataset to avoid overfitting, however this was for structured text based data in pandas using simple models from sklearn. Is this still the case for image based datasets that will be trained on a CNN? I have been trying to oversample only the train data by applying augmentations to the images. However, for some reason I get a train accuracy of 1.0 and a validation accuracy of 0.25 on the very first epoch, where the numbers dont really change as the epochs progress which doesn't make sense to me. Should the oversampling be applied to the entire dataset and should the image augmentations be applied to only the new data or all of it?

My dataset is of RGB landscape images with 7 different classes, without about 29k total images. Doing a 70-15-15 split there are about 20k train samples before oversampling

Answer 1

It is unlikely that you should oversample in any of these cases. Almost all motivation for oversampling comes from evaluating models using classification accuracy. Appealing as classification accuracy first seems, it turns out to be highly problematic and hides many assumptions that might be, and often are, hideously inappropriate. (Metrics like sensitivity, specificity, and $F_1$ suffer from many of the same issues.)

Proper statistical methods that evaluate the continuous predictions of models, which often have an interpretation as probability, provide much richer information than the forces classifications, and such methods handle class imbalance with minimal trouble. (The (interesting) answer to the linked question about class imbalance and oversampling is really a matter of experimental design.) Standard examples of such statistical methods are evaluation of the crossentropy loss or Brier score.

Answer 2

Oversampling should still only be applied to the training data, even for image-based datasets. The reason for this is the same as with structured data: you want to avoid leaking information from the test/validation set into the training set, which could lead to overfitting.

The issue you're experiencing with the training accuracy being 1.0 and the validation accuracy being 0.25 from the first epoch might be due to several reasons:

1. Overfitting: If you're oversampling too much, your model might be overfitting to the training data. This would explain why the training accuracy is perfect but the validation accuracy is low.

2. Data leakage: If there's any overlap between your training and validation sets, your model might be memorizing the training data, leading to a high training accuracy but low validation accuracy.

3. Incorrect validation set: If your validation set is not representative of your training data, your model might perform poorly on it even if it performs well on the training data.

4. Model complexity: If your model is too complex, it might be overfitting to the training data. Try using a simpler model or adding regularization to prevent overfitting.

As for the image augmentations, they should be applied to the new data generated by oversampling. The purpose of augmentations is to create new, slightly different versions of your existing images to increase the diversity of your training data and prevent overfitting. Applying augmentations to all of your data, including the original images, might lead to overfitting as your model could learn to recognize the specific augmentations instead of the underlying patterns in the images.