0
$\begingroup$

I would like to train different machine learning algorithms (SVM, Random Forest, CNN etc.) for the same data set (e.g. MNIST) und then compare their accuracies. The goal would be to find out from which training data size which method is preferable to the others. To do this I continuously reduce the original training data set (of 60000 samples) and train the models on these reduced trainings data sets.

If I then determine the accuracy using the original MNIST-test dataset (10000 samples), of course I will get overfitting, e.g. with a training data set of 1000 samples I get a training accuracy of 95% and test accuracy of 75%.

The smaller the training data set, the lower the test accuracy, while the training accuracy remains at about the same level.

Would it make sense also to reduce the test data set to restore the original 1:6 ratio of the test set : training set? Personally, I think that does not make sense. Or have I thought incorrectly about that?

$\endgroup$
  • $\begingroup$ Hi @Code Now! In short- it doesn't make sense, so your thought about it is correct. If you elaborate your thought I can evaluate it and extend if needed. $\endgroup$ – wind Apr 29 at 8:43
  • $\begingroup$ Hi @wind, thank you for your reply. At present, one problem is that when I training a model with e.g. 1000 training samples instead of 60000 then I get significant overfitting, even if I optimize the hyperparameters e.g. through GridSearch. At the moment I see only one possibility to reduce/prevent the overfitting as long as possible by extending the training dataset by data augmentation. $\endgroup$ – Code Now Apr 29 at 10:07
0
$\begingroup$

It's normal (and expected even) to have a Test Set that is smaller than your Training Set.

In general, the more training data you have, the better your performance should be. That is, there'll be less variation in your model parameters if trained on more examples.

This is especially true for Deep Learning approaches (your Convolutional Neural Network, CNN, you mentioned).

Below is a graph that illustrates the above. Where 'Classical Learning' refers to Machine Learning models (SVM, Random Forest, etc.)

enter image description here

From this explanation and graph, it should now make sense why you get a lower test accuracy when you have a smaller training set. There simply isn't enough variation in the training samples, leading to overfitting.

There are numerous ways to avoid overfitting, through data augmentation, adding Dropout layers (in the case of CNN), or regularizing your model. Going into the details is beyond this answer.

However, one of the best and easiest ways to help avoid overfitting is through increasing the the training set.

By increasing the training set, you follow the graph above and decrease variation in parameters but more importantly expose your model to more samples and variation in the dataset.

This helps explain, in part, why you see a higher test accuracy when using a larger test set.

Should you reduce your test set to maintain a ratio? Probably not, in a similar way to your parameters experiencing variation with a small training set the same is true with your test set. You'll want a big enough test set to cover enough variation in samples so the accuracy returned can be trusted.

This StackOverflow answer provides good detail and rationale on choosing the training/test split.

Let me know if you have any questions! I can't comment yet, hence a relatively general answer.

$\endgroup$
0
$\begingroup$

There is no reason to sub-sample your test data. The test data serves to give an unbiased estimation of your model learner's performance on unseen data, and more test data only gives you a more accurate estimate. You should test on as much data as possible. Normally this comes as a tradeoff when splitting between training and testing data (more test data means less training data, which is arguably more important), but in this case you are purposefully reducing the training set size to analyze the effect of that reduction. In this case, having more test data does not come at the cost of less training data, so there's no downside to using them for testing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.