In my problem, I am dealing with a highly imbalanced data set, say for every positive class there are 10000 negative one. A normal starting method to train a model is to undersample the data. In this procedure, it is very important the train our model on the undersampled data and check the model evaluation on the holdout (from the original data -- without undersampling).
Now the quesion is here. KFold-cross validation actually splits the undersampled train set into K segments and take one of the folds as test set (which is now undersampled test set). I believe for model evaluation we actually need to calculate the metric of interest on non-undersampled test set (right? of I am misunderstaning sth here?). If yes, is it possible to perfrom cross validation as follows:
- Split data into K segments.
- Take the first Segment as test set, and undersample the rest of Folds (for example K=1 as test and K=2,3,4,5 as train sets)
- Fit model on undersampled train data and calcualte the metric of interest on the test set.
- Consider the other Fold as test set (this time e.g., K=2) and the rest as train set (K=1,3,4,5). Undersample train set and proceed to step 3.
- Continue this procedure for the rest of the folds.
Is this a correct way of cross validation, when we undersample the data? If yes, is possible to do it with standard libraries?