Most data imbalance questions on this stack have been asking How to learn a better model, but I tend to think one other problem is How do we define "better" (i.e. fairly evaluate the learned model) to ensure the evaluation performance on the limited test set does not suffer from high variance, since many of the imbalance classes also have limited number of samples. I'll summarize my current understanding below. First let's suppose we have 1000 samples, among which only 10 have positive labels. Our task is binary classification.
One common practice is to do cross-validation. In a typical cross-validation setting, one split the data to remaining-holdout first, and then conduct cross-validation on the remaining set to train-val folds for hyperparameter tuning. Evaluation of the model is performed on the test set, by either using one of the trained fold models, ensemble of all folds, or re-trained model on the full remaining set. However, the test set is still comprised of only two positive samples, which might very likely suffer from variance in the split.
Built upon such intuition, one natural thought is to do three-way cross-validation, which splits each fold to train-val-test sets. The test performance would be the average of all test performances across folds. However, it really confuses me because we are not even using one single model for evaluation, and it cannot really tell us how the one selected model would behave in the real world.
I understand that we have the assumption that test distribution should be the same as the general. But two test positive samples are not able to capture a distribution. In such scenario, what should we do?
