I have a 3 to 1 imbalanced set. I'm using KFold for cross validation. For training I'm using RandomOverSampler to balance the training set, but the validation set is left as is. I'm testing different models at the same time (RandomForest, SVC, GradientBoosting, XGBoost)
However, I'm unsure whether this is dealing with the imbalance.
I check F1 macro score and F1 scores for both labels using the pos_label parameter. I notice that on the training set, these values are similar i.e. 0.85, 0.84 and 0.86. But on the validation set, they are quite apart, i.e. 0.71, 0.64 and 0.79. I would expect that both F1 scores would be balanced, but it doesn't seem like it. All the models perform in a similar fashion.
Should I be doing this?
Follow ups, is the model learning properly or not? How to evaluate this in this case?
