How much data should we use during training, and how much in testing? Can anyone explain why does it always seem to be 70:30 or 80:20 ratios?
$\begingroup$
$\endgroup$
1
-
3$\begingroup$ See here for a similar question. Note that cross validation is much more useful. $\endgroup$– StereoCommented Mar 3, 2017 at 9:19
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
Those rule of thumbs for ratio make no sense. The use case for a test set is to measure your performance in a real scenario on unseen data. How much you need for an accurate measure depends on the level of accuracy you need, the amount of variance you expect, not so much on the size of the original dataset. Something to be aware of is that you do decrease the size of your training set, which is a cost of testing. But when you have 10 million rows of data it makes no sense to validate on 3 million, when 10,000 could be more than enough. Outside of the cost of losing training data you should look at absolute numbers in your test set.
answered Mar 3, 2017 at 10:18
$\begingroup$
$\endgroup$
As far as reason behind train/test classification is concerned..you can have some help here
$\begingroup$
$\endgroup$
You and others are correct to question these "rules". Truth is, choosing a test/training split is a simple question with a complicated answer. See this paper which discusses different cross-validation methodologies, where
As a conclusion, guidelines are provided for choosing the best cross-validation procedure according to the particular features of the problem in hand.
In particular, they found the signal-to-noise ratio of the data had a lot to do with how the model should be trained and tested.
You can also find empirical papers (here's my dataset, here's my goal, here's what worked best) such as this one which found 10-fold cross-validation was best for model selection with that particular data set.
While I don't claim to have read everything, I haven't yet stumbled upon some who found a simple 80/20 or 70/30 split was the best way to do anything.
$\begingroup$
$\endgroup$
6
The proportion of your data split is arbitrary.
Things to remember however: the more training data you have, the better your model will be. The more testing data you have, the less variance you can expect in your results (ie. accuracy, false positive rate, etc.).
Ideally if you had infinite training data, you would be able to maximize the model's performance. The model would be the best it would ever be. And if you had infinite testing data you will be 100% confident in the resulting accuracy of your model.
Now, if you do not have sufficient testing data, for example you only have 1 example. Then it is possible that for the one example your algorithm correctly identifies its class and thus you assume that you have a 100% accuracy. But that would be wrong. You need multiple testing examples in order to correctly assess the performance of your model.
The 80:20 split is popular. As well as the 90:10. It's arbitrary. It all depends on how much data you have at hand. It also depends on how much data you expect to be sufficient to accurately train your model. If you only have 100 examples and you are training a data intensive model such as an NN then a 90:10 split is probably better. Although you will have high variance in your accuracy but your model will generalize better due to it having more data to train with.
I usually stick to 80:20 unless the dataset suggests otherwise. Also, it is good practice to use a validation set. If you have sufficient data, it is better to do 60:20:20. Train on 60% of the data, validate your model and tweek it on 20% of the data and when you are ready to submit your model test it on the final 20% of the data.
Cross-validation is also a good technique. You split your data into n bins. You then do leave-one-out training. You train on all the data bins except for 1, use the remaining bin to test. Repeat this procedure n times and take the average of your accuracies. Be wary however that you are spoiling some of your data by doing this since you are making use of all the data. It's always better to keep a chunk of the data for the final product, untouched until the last minute.
-
$\begingroup$ It's hardly an arbitrary choice. Each technique and partitioning strategy has consequences, and many are sub-optimal. $\endgroup$– PeteCommented Mar 7, 2017 at 17:39
-
$\begingroup$ Sometimes a general rule of thumb is best. You can delve into the specifics of every single arbitrary aspect of a project but then you become a burden. Generally, if you are given a data set and you wish to estimate the performance of your trained model, a 80:20 split works very well. Read the entirety of my comment, I clearly state that their are cases where this would not work, I also provided an additional method, cross-validation, which may be suitable. Thanks for the downvote albeit you not providing any suitable answer to the problem nor fully understanding the answer I provided. $\endgroup$– JahKnowsCommented Mar 8, 2017 at 15:10
-
$\begingroup$ Can you link to any published research (not a blog) which supports your claim-- that a single split works well, and the amount of the split is arbitrary? In that case I will apologize and change my vote. $\endgroup$– PeteCommented Mar 8, 2017 at 16:31
-
$\begingroup$ This isn't something that is usually published in papers because of its arbitrary nature. It is heavily dependent on the nature of the dataset or the amount of available data. However, I usually point students to these two starting resources: coursera.org/learn/ml-foundations/lecture/DHRlE/… $\endgroup$– JahKnowsCommented Mar 8, 2017 at 16:51
-
$\begingroup$ and a book that teaches how to implement a NN and CNN from scratch: neuralnetworksanddeeplearning.com/chap1.html. Moreover, notice that the most famous database ever used in Machine Learning model testing is the MNIST dataset. This set uses a 86% and 14% split. $\endgroup$– JahKnowsCommented Mar 8, 2017 at 16:53