Splitting the dataset manually for k-Fold Cross-Validation

Question

I manually divided the dataset into three sets: train, test, and validation. Each set includes several folders, one for each patient. Each patient has many images from a different point of view. As a result, I manually divided the dataset by patient folders to avoid having the same patient appear in more than one set.

Train:
   class 1:
      patient_1:
         a.png
.......

Now I'd like to apply k-Fold Cross-Validation on a manually split dataset. Is it possible to do so?.

x_train,y_train= load_mydata()    
x_test,y_test= load_mydata()
x_val,y_val= load_mydata()

from sklearn.model_selection import cross_val_score
    # evaluate model
scores = cross_val_score(model, ?, ?, scoring='accuracy', cv=cv, n_jobs=-1)

Can I re-split the dataset to 50% for training and 50% for testing and using them in the cross validation two times?

Answer 1

It depends why you want to use cross-validation (CV). CV is meant to provide a more reliable performance estimation.

• It can be used instead of splitting between training and test set. In this case you provide the full dataset and the CV process splits it randomly $k$ times. If you just want the performance of one model this is fine, but you should not test several models or methods in this way.
• It can be used for selecting the best model/method out of multiple options. In this case:
  • the CV process should be applied only on the training set,
  • then the best model/method is selected (and usually retrained on the full training data)
  • only the final model is applied to the test data.