I have a severely skewed data sets consisting of 20 something classes where the smallest class contains on the order of 1000 samples and the largest several millions.
Regarding the validation data, I understand that I should make sure that it represent a similar ratio between classes compared to the one in my original raw data. Hence, I shouldn't do any under- or over-sampling on that validation data, but can do it on the training data.
Because I have such greatly skewed data set, is it still viable to add some restriction to the selection of my validation data set? Say I want there to be at least 1000 samples from each class in order to accept it, as I want to have a reasonable accuracy on the metrics of all classes.
Would this ruin my validation as the ratio between the largest and smallest class could then go from ~0.01-0.1% to ~1.0%, or is it still safe as the validation data still is significantly skewed?