I manually divided the dataset into three sets: train, test, and validation. Each set includes several folders, one for each patient. Each patient has many images from a different point of view. As a result, I manually divided the dataset by patient folders to avoid having the same patient appear in more than one set.

   class 1:

Now I'd like to apply k-Fold Cross-Validation on a manually split dataset. Is it possible to do so?.

x_train,y_train= load_mydata()    
x_test,y_test= load_mydata()
x_val,y_val= load_mydata()

from sklearn.model_selection import cross_val_score
    # evaluate model
scores = cross_val_score(model, ?, ?, scoring='accuracy', cv=cv, n_jobs=-1)

Can I re-split the dataset to 50% for training and 50% for testing and using them in the cross validation two times?


It depends why you want to use cross-validation (CV). CV is meant to provide a more reliable performance estimation.

  • It can be used instead of splitting between training and test set. In this case you provide the full dataset and the CV process splits it randomly $k$ times. If you just want the performance of one model this is fine, but you should not test several models or methods in this way.
  • It can be used for selecting the best model/method out of multiple options. In this case:
    • the CV process should be applied only on the training set,
    • then the best model/method is selected (and usually retrained on the full training data)
    • only the final model is applied to the test data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.