What could be the possible reasons for a significant difference in cross validation and testing f1_scores? I am performing 3 fold Stratified cross validation and the testing f1_score is almost 0.15 less than cross validation score. How can I come up with a more effective cross validation strategy so that the two scores are closer?
Significant differences between the calculated classification performance in cross-validation and in the final test set appear obviously, when the model is overfitted.
A good indicator for bad (i.e., overfitted) models is a high variance in the F1-results of single iterations in the cross-validation.
Possible strategies to get a better estimation of the model performance would be:
- useing more folds (e.g., 10-fold cross-validation, or leave-one-out cross-validation)
- considering simpler models (e.g., less variables, more general parameters)
- considering other machine learning algorithms
There can be many reasons for this thing but in most of the cases I have observed one common reason. When you split your data using train_test_split or any other method, it is important to note that the column on which you are splitting the data is significant for splitting considering both the training and testing set. For example if I have a 'time' field in my data and I have split the data into training and testing sets on this column such that there is no value in this column of the test set which matches with any of the values in the same columns in the train set.