I would like to train different machine learning algorithms (SVM, Random Forest, CNN etc.) for the same data set (e.g. MNIST) und then compare their accuracies. The goal would be to find out from which training data size which method is preferable to the others. To do this I continuously reduce the original training data set (of 60000 samples) and train the models on these reduced trainings data sets.
If I then determine the accuracy using the original MNIST-test dataset (10000 samples), of course I will get overfitting, e.g. with a training data set of 1000 samples I get a training accuracy of 95% and test accuracy of 75%.
The smaller the training data set, the lower the test accuracy, while the training accuracy remains at about the same level.
Would it make sense also to reduce the test data set to restore the original 1:6 ratio of the test set : training set? Personally, I think that does not make sense. Or have I thought incorrectly about that?