I am building a binary classification model for imbalanced data (e.g., 90% Pos class vs 10% Neg Class).
I already balanced my training dataset to reflect a a 50/50 class split, while my holdout (training dataset) was kept similar to the original data distribution (i.e., 90% vs 10%). My question is regarding the validation data used during the CV hyperparameter process. During each iteration fold should:
1) Both the training and test folds be balanced
2) The training fold should be kept balanced while the validation fold should be made imbalanced to reflect the original data distribution and holdout dataset.
I am currently using the 1st option to tune my model; however, is this approach valid given that the holdout and validation datasets have different distributions?