Restrictions on my skewed validation data

Question

I have a severely skewed data sets consisting of 20 something classes where the smallest class contains on the order of 1000 samples and the largest several millions.

Regarding the validation data, I understand that I should make sure that it represent a similar ratio between classes compared to the one in my original raw data. Hence, I shouldn't do any under- or over-sampling on that validation data, but can do it on the training data.

Because I have such greatly skewed data set, is it still viable to add some restriction to the selection of my validation data set? Say I want there to be at least 1000 samples from each class in order to accept it, as I want to have a reasonable accuracy on the metrics of all classes.

Would this ruin my validation as the ratio between the largest and smallest class could then go from ~0.01-0.1% to ~1.0%, or is it still safe as the validation data still is significantly skewed?

Answer 1

I suggest using your whole validation set but providing class-specific metrics. E.g., AUC only considering validation rows with targets in a single class. That will help you see if you're underperforming for a specific class. You can then say "This model earned 0.xxx AUC on the whole validation set and no worse than 0.yyy for any single class."

Answer 2

It is best practice to not make any restrictions or manipulations just on a validation dataset, this includes removing categories or downsampling.

Changing just the validation dataset has the potential to negatively impact the ability to generalize. For example, if you manipulate your validation dataset and then pick hyperparameter values on the changed validation dataset, those hyperparameters have less of a chance to generalize to other datasets.

Restrictions or manipulations to the data should happen before splitting into train and validation datasets.

The Elements of Statistical Learning by Hastie et al. in section 7.10.2 "The Wrong and Right Way to Do Cross-validation" goes into greater detail.