I am trying to classify images to more then a 100 classes, of different sizes ranged from 300 to 4000 (mean size 1500 with std 600). I am using a pretty standard CNN where the last layer outputs a vector of length number of classes, and using pytorch's loss function CrossEntropyLoss.
I tried to use $weights = \frac{max(sizes)}{sizes}$ for the cross entropy loss which improved the unweighted version, not by much.
I also thought about duplicating images such that all classes ends up to be of the same size as the larges one.
Is there any standard way of handling this sort of imbalance?