Please I am a bit confused. I am doing some practice work.

I am using the F- score to score model's performance.

To improve model's performance, I did a random grid search and got an F-score of 1. To check the performance of the model on my validation set, I used the estimator with the optimal parameter values and got an F-score of about 0.957, but when I tried to check its performance on the test data, I got an F-score of about 0.86.

Not satisfied with this I ran a 10 K-fold cross validation and got a mean and standard deviation of 0.96 and 0.05 respectively. The cross validation scores ranged from about 0.83 to 0.96

My question is: Is my model overfitting judging from the F-score from the test data, since it seems not to, though slightly, on the validation set.

Thank you.


Edit: My dataset has about 95000 samples and a very high class imbalance (99.8% /0.02%). My aim is to predict the minority class.

I split the original data into training and testing sets (0.65/0.35), then from the test I split into half, getting another test and validation set.

The parameters of the RF model i ran a random grid search were n_estimators, max depth, max leaf nodes, and max features and got: n_estimators = 250, max leaf nodes = 60, max features= 13, and max depth 26, as best parameters.

The 10K-fold CV result was 0.906 not 0.96 for the mean, 0.05 for standard dev.

  • $\begingroup$ Hello Gozie, in order to properly answer your question, can you provide more information about your data? What are the exactly sizes of training, test, and validation sets? Which parameters of your RF are you changing? $\endgroup$ – Victor Oliveira Sep 10 '19 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.