Ive build a RF model for an imbalanced data set that after feature selection has an F1 score of 54.26%. I am now trying to do hyper parameter tuning using RandomizedSearchCV, after creating validation curve plots for each hyper parameter to identify a more promising grid.

However, when I create the validation curve plots and execute the RandomisedSearchCV both using train data only, there is a huge difference between the train- and validation set (Cross validation "test set") f1 score performance: train f1 score is 99%+ and validation f1 score is close towards test data set f1 score). This is something I would except since I am training on the train data and then predicting on it (basically giving it the answers). Yet when I see other examples online their train- and validation scores lie very close together and they also do the validation curve plots/RandomizedSearchCV on train data only. Anyone has any idea what I could be doing wrong? Many thanks in advance!

  • $\begingroup$ This means that your model overfits. Are you using any kind of oversampling/undersampling method in order to deal with the imbalance by any chance? $\endgroup$ – Erwan Jan 22 '20 at 1:34
  • $\begingroup$ What hyperparameter space/optimum? Which "other examples online"? $\endgroup$ – Ben Reiniger Jan 22 '20 at 12:34
  • $\begingroup$ I am using under sampling, however I turned this off during RandomizedSearchCV and/or plotting the validation curves as I noticed validation F1 score became not representative anymore of the actual test F1 score. $\endgroup$ – 19dr95 Jan 22 '20 at 12:37
  • $\begingroup$ towardsdatascience.com/… . This is the example I mean, although accuracy is used here as test metric. $\endgroup$ – 19dr95 Jan 22 '20 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.