Ive build a RF model for an imbalanced data set that after feature selection has an F1 score of 54.26%. I am now trying to do hyper parameter tuning using RandomizedSearchCV, after creating validation curve plots for each hyper parameter to identify a more promising grid.
However, when I create the validation curve plots and execute the RandomisedSearchCV both using train data only, there is a huge difference between the train- and validation set (Cross validation "test set") f1 score performance: train f1 score is 99%+ and validation f1 score is close towards test data set f1 score). This is something I would except since I am training on the train data and then predicting on it (basically giving it the answers). Yet when I see other examples online their train- and validation scores lie very close together and they also do the validation curve plots/RandomizedSearchCV on train data only. Anyone has any idea what I could be doing wrong? Many thanks in advance!