In a case of imbalanced data classification, I know that we only oversample the training set (to prevent data leakage from training to testing subsets), but what if there are no positive data points in my testing set? The testing set is still highly skewed and has only 1% of my positive class. I am using XGBoost, Random Forest, Logistic Regression, and KNN for the classification task.
Also, I have tried SMOTE, SMOTE-NC, and Class_weight to oversample my training set. To increase the chance of having more data from the minority class, I changed the 10-fold to 5-fold cross-validation (when developing the models), no improvement!
PS: I have >100K data points in my dataset.