I'm a beginner in machine learning and I have a special case in which I have only a small training dataset of about 500 images and a test dataset of 10,000 images. Does it still make sense to do a 10-fold-cross-validation or Repeated-Cross-validation on the training data? Or would this be not necessary anymore due to the large test dataset?

Many thanks in advance


Generally speaking, when evaluating a model you either choose to do cross-validation or train-test splitting, but not both. Your dataset appears to already be split between training and testing sets, so you seem to have implicitly chosen train-test splitting.

In your particular case, it may still make sense to run cross-validation if you have a reason to mistrust the results from the testing set. Is the distribution of labels in the test data balanced or highly skewed? Is the test data representative of examples you're likely to see in production? If your test data is representative, then you'll likely get a better accuracy estimate evaluating on the test set rather than running cross-validation.

But I'm confused. How did you come to have a training set of 500 images and testing set of 10,000 images? If you 10,500 labeled examples available to you, then you can divide them however you like - right? Why not run cross-validation with the entire set of 10,500 images? That will give you the most robust accuracy estimate.

  • $\begingroup$ It is the MNIST dataset. The 500 images are a subset of a training dataset that consists of 60,000 images. These 500 images were taken at random from the full training dataset and considering the distribution of the classes. I randomly determined this subset by using train_test_split with stratify=y. It is about an investigation which classifiers achieve the best accuracy (classification performance) with such a few training dataset. $\endgroup$ – Code Now Oct 8 at 13:37
  • $\begingroup$ Ah, I see. If it's MNIST, then you should evaluate with the testing set. Don't worry about cross validation. The MNIST testing set was constructed to be representative. $\endgroup$ – zachdj Oct 8 at 14:05
  • 2
    $\begingroup$ Thank you very much. Then I will test the models only with the test dataset. But I could use the cross validation to optimize the hyperparameters for the small training dataset via GridSearchCV. Since I assume a situation in which in fact only 500 training images are available. $\endgroup$ – Code Now Oct 8 at 14:12
  • $\begingroup$ Yep! That's a good idea. $\endgroup$ – zachdj Oct 8 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.