I have a severely skewed data sets consisting of 20 something classes where the smallest class contains on the order of 1000 samples and the largest several millions.

Regarding the validation data, I understand that I should make sure that it represent a similar ratio between classes compared to the one in my original raw data. Hence, I shouldn't do any under- or over-sampling on that validation data, but can do it on the training data.

Because I have such greatly skewed data set, is it still viable to add some restriction to the selection of my validation data set? Say I want there to be at least 1000 samples from each class in order to accept it, as I want to have a reasonable accuracy on the metrics of all classes.

Would this ruin my validation as the ratio between the largest and smallest class could then go from ~0.01-0.1% to ~1.0%, or is it still safe as the validation data still is significantly skewed?

  • $\begingroup$ Hi there, you can also go for Sampling during training time to generalize a bit better $\endgroup$
    – Aditya
    Jun 2 '18 at 1:46

I suggest using your whole validation set but providing class-specific metrics. E.g., AUC only considering validation rows with targets in a single class. That will help you see if you're underperforming for a specific class. You can then say "This model earned 0.xxx AUC on the whole validation set and no worse than 0.yyy for any single class."


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.