i have imbalanced data consisting of nine classes, and i am planning to collapse them into two classes. i performed stratified (proportionate) sampling between test, validation, and training sets according to the nine classes. Now, when i oversample the training data, should i oversample the nine classes before collapsing them, or should collapse them first and then oversample?
Generally speaking, i believe oversampling then collapsing would be better, but some of the classes in the training data are quite small (5 instances) while others are quite large (1000 instances). Hence, i will be repeat sampling the same 5 instances 1,000, which seems odd. On the other hand, if i collapse then oversample, then there is a good chance the smallest classes may not even end up being resampled.
any advice? thanks!