What could be the possible reasons for a significant difference in cross validation and testing f1_scores? I am performing 3 fold Stratified cross validation and the testing f1_score is almost 0.15 less than cross validation score. How can I come up with a more effective cross validation strategy so that the two scores are closer?

  • 1
    $\begingroup$ Worth explaining what your current cross-validation and testing strategies are. Also, is the test set under your control, or is it held back by an organiser (e.g. Kaggle, ImageNet competition)? This is important, since you will want to match the organiser's train/test sampling strategy, amongst other concerns. $\endgroup$ – Neil Slater Aug 30 '16 at 12:24
  • $\begingroup$ What's your classifier? What type of problem is it (text? timeseries? search ranking? etc.) Is there a class imbalance, if so how much? What happens if you try K= 5/10-fold crossvalidation? Did you tune your classifier's hyperparameters to the training set (if so, that's not recommended)? How many rows(exemplars) and columns (features) in your training and test sets? $\endgroup$ – smci Sep 8 '16 at 19:15
  • 1
    $\begingroup$ @smci the problem type was time series. I tuned parameters to the validation set. The discrepancy was most likely due to the testing data belonging to a different time frame. $\endgroup$ – banad Oct 2 '16 at 8:21
  • $\begingroup$ @banad: Ok. Please post that as an answer. $\endgroup$ – smci Oct 3 '16 at 0:07

Significant differences between the calculated classification performance in cross-validation and in the final test set appear obviously, when the model is overfitted.

A good indicator for bad (i.e., overfitted) models is a high variance in the F1-results of single iterations in the cross-validation.

Possible strategies to get a better estimation of the model performance would be:

  • useing more folds (e.g., 10-fold cross-validation, or leave-one-out cross-validation)
  • considering simpler models (e.g., less variables, more general parameters)
  • considering other machine learning algorithms

There can be many reasons for this thing but in most of the cases I have observed one common reason. When you split your data using train_test_split or any other method, it is important to note that the column on which you are splitting the data is significant for splitting considering both the training and testing set. For example if I have a 'time' field in my data and I have split the data into training and testing sets on this column such that there is no value in this column of the test set which matches with any of the values in the same columns in the train set.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.