All data in my binary classification problem are represented by X and y. Now, I do stratified cross-validation on this data as follows:

scoring = {'accuracy' : make_scorer(accuracy_score), 
           'precision' : make_scorer(precision_score),
           'recall' : make_scorer(recall_score), 
           'f1_score' : make_scorer(f1_score)}


results = cross_validate(estimator=model, X=X, y=y, cv=10, scoring=scoring)

If run the code, I will get the following results:

Accuracy : 0.5436815489342804
Precision : 0.020165565854870747
Recall : 0.11013513513513513
F1_score : 0.03315023853741518

Now, I split X and y as follows:

#Test, training data split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, stratify = y)

#Split the training data into validation set 
X_val_train, X_val_test, y_val_train, y_val_test = train_test_split(X_train, y_train, test_size = 0.1, random_state=0, stratify=y_train )

Now, if I do the same cross-validation procedure like before on X_train and X_train, I will get the following results:

Accuracy : 0.8424393681243558
Precision : 0.47658195862621017
Recall: 0.1964997354963851
F1_score : 0.2773991741912054

I do not understand why results are so different and why this happened.

  • $\begingroup$ Possibly because the default in cross_validate is to not shuffle samples within each class, whereas train_test_split does shuffle the data? Try kf=StratifiedKFold(10, shuffle=True), and pass cv=kf in the cross_validate? $\endgroup$ – Ben Reiniger Mar 26 at 20:48
  • $\begingroup$ At first sight I can't think of a good reason why this would happen. I assume that there are few positive instances, right? Can you please add the details: how many instances in total and by class? $\endgroup$ – Erwan Mar 26 at 21:09
  • $\begingroup$ In total there are 9190 samples with 1376 from class 1 and the rest from class two. When I split data, there are 7352 samples and 1101 ones from class 1. $\endgroup$ – Katatonia Mar 26 at 23:51
  • $\begingroup$ Ok it's not that imbalanced. I think there's a good chance that there's a bug in your code somewhere. Can you try (1) to do both variants in the same program, this way you make sure that it's exactly the same data used; you should call the first model model1 and the second model2 to avoid any confusion. (2) try to apply model1 to the training data in variant 1 and same for model2 with variant2; (3) if the bug hasn't appeared yet and the results are still the same, try to apply model1 to the test data of model2 and conversely... $\endgroup$ – Erwan Mar 27 at 0:06
  • $\begingroup$ ... the last experiment is not meant to do anything meaningful, it's just to try to observe where something goes wrong. If results are still weird please edit the question so that we can see all these results. $\endgroup$ – Erwan Mar 27 at 0:08

Without having more information my hypothesis the complexity of your model.

I think you might have a model so complex that it needs a "large" amount of data for having a good performance (First scenario with cross validation on full data)

Then when you split your data in train - test set and repeat the cross validation the amount of data is reduced naturally so it is causing that your model do not have enough samples to generalize.

That might be confirmed via a learning curve:

enter image description here

If the training and cross-validation scores converge together as more data is added (shown in the left figure), then the model will probably not benefit from more data. If the training score is much greater than the validation score then the model probably requires more training examples in order to generalize more effectively.


In the first way, you are looking all the data.

The second way splitting makes a split with a fixed seed. It appears that slice of data happens to be very different than all the data.

  • $\begingroup$ I have used different seeds or even random seed but the same results obtained. $\endgroup$ – Katatonia Mar 26 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.