Please I am a bit confused. I am doing some practice work.
I am using the F- score to score model's performance.
To improve model's performance, I did a random grid search and got an F-score of 1. To check the performance of the model on my validation set, I used the estimator with the optimal parameter values and got an F-score of about 0.957, but when I tried to check its performance on the test data, I got an F-score of about 0.86.
Not satisfied with this I ran a 10 K-fold cross validation and got a mean and standard deviation of 0.96 and 0.05 respectively. The cross validation scores ranged from about 0.83 to 0.96
My question is: Is my model overfitting judging from the F-score from the test data, since it seems not to, though slightly, on the validation set.
Edit: My dataset has about 95000 samples and a very high class imbalance (99.8% /0.02%). My aim is to predict the minority class.
I split the original data into training and testing sets (0.65/0.35), then from the test I split into half, getting another test and validation set.
The parameters of the RF model i ran a random grid search were n_estimators, max depth, max leaf nodes, and max features and got: n_estimators = 250, max leaf nodes = 60, max features= 13, and max depth 26, as best parameters.
The 10K-fold CV result was 0.906 not 0.96 for the mean, 0.05 for standard dev.