Smaller test data set than training data set in machine learning

Question

I would like to train different machine learning algorithms (SVM, Random Forest, CNN etc.) for the same data set (e.g. MNIST) und then compare their accuracies. The goal would be to find out from which training data size which method is preferable to the others. To do this I continuously reduce the original training data set (of 60000 samples) and train the models on these reduced trainings data sets.

If I then determine the accuracy using the original MNIST-test dataset (10000 samples), of course I will get overfitting, e.g. with a training data set of 1000 samples I get a training accuracy of 95% and test accuracy of 75%.

The smaller the training data set, the lower the test accuracy, while the training accuracy remains at about the same level.

Would it make sense also to reduce the test data set to restore the original 1:6 ratio of the test set : training set? Personally, I think that does not make sense. Or have I thought incorrectly about that?

Answer 1

It's normal (and expected even) to have a Test Set that is smaller than your Training Set.

In general, the more training data you have, the better your performance should be. That is, there'll be less variation in your model parameters if trained on more examples.

This is especially true for Deep Learning approaches (your Convolutional Neural Network, CNN, you mentioned).

Below is a graph that illustrates the above. Where 'Classical Learning' refers to Machine Learning models (SVM, Random Forest, etc.)

From this explanation and graph, it should now make sense why you get a lower test accuracy when you have a smaller training set. There simply isn't enough variation in the training samples, leading to overfitting.

There are numerous ways to avoid overfitting, through data augmentation, adding Dropout layers (in the case of CNN), or regularizing your model. Going into the details is beyond this answer.

However, one of the best and easiest ways to help avoid overfitting is through increasing the the training set.

By increasing the training set, you follow the graph above and decrease variation in parameters but more importantly expose your model to more samples and variation in the dataset.

This helps explain, in part, why you see a higher test accuracy when using a larger test set.

Should you reduce your test set to maintain a ratio? Probably not, in a similar way to your parameters experiencing variation with a small training set the same is true with your test set. You'll want a big enough test set to cover enough variation in samples so the accuracy returned can be trusted.

This StackOverflow answer provides good detail and rationale on choosing the training/test split.

Let me know if you have any questions! I can't comment yet, hence a relatively general answer.

Answer 2

There is no reason to sub-sample your test data. The test data serves to give an unbiased estimation of your model learner's performance on unseen data, and more test data only gives you a more accurate estimate. You should test on as much data as possible. Normally this comes as a tradeoff when splitting between training and testing data (more test data means less training data, which is arguably more important), but in this case you are purposefully reducing the training set size to analyze the effect of that reduction. In this case, having more test data does not come at the cost of less training data, so there's no downside to using them for testing.