16
$\begingroup$

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining.

My colleague (who is a machine learning expert) said that, in his opinion, the arguably most important practical aspect of machine learning is how to understand whether you have collected enough data to train your machine learning model.

This statement surprised me, because I had never given that much importance to this aspect...

I then looked for more information on the internet, and I found this post on FastML.com reporting as rule of thumb that you need roughly 10 times as many data instances as there are features.

Two questions:

1 - Is this issue really particularly relevant in machine learning?

2 - Is the 10 times rule working? Are there any other relevant sources for this theme?

Improve this question
$\endgroup$
5
  • $\begingroup$ 1. Yes. 2. It's a good baseline but you can get around it with regularization to reduce the effective degrees of freedom. This works especially well with deep learning. 3. You can diagnose the situation on your problem by plotting the learning curve of the sample size against the error or score. $\endgroup$
    – Emre
    Commented Jun 26, 2017 at 21:58
  • $\begingroup$ @Emre Thanks! Can you also suggest me some papers or any material to read? $\endgroup$
    – DavideChicco.it
    Commented Jun 28, 2017 at 2:14
  • $\begingroup$ This will usually be covered alongside cross-validation and other model validation techniques in your textbook. $\endgroup$
    – Emre
    Commented Jun 28, 2017 at 4:32
  • $\begingroup$ The 10 times rule is great if you can achieve it, but it is just not practical in some business settings. There are many situations where the number of features is much greater than data instances (p>>n). There are machine learning techniques designed specifically to deal with these situations. $\endgroup$
    – acylam
    Commented Aug 29, 2017 at 15:59
  • $\begingroup$ If you need a detailed explanation which can help you to understand the learning curve graph check this out: scikit-yb.org/en/latest/api/model_selection/learning_curve.html $\endgroup$
    – shrikanth singh
    Commented Nov 11, 2019 at 17:11

3 Answers 3

Reset to default
8
$\begingroup$

The ten times rule seems like a rule of thumb to me, but it is true that the performance of your machine learning algorithm may decrease if you do not feed it with enough training data.

A practical and data-driven way of determining whether you have enough training data is by plotting a learning curve, like the one in the example below:

Learning curve

The learning curve represents the evolution of the training and test errors as you increase the size of your training set.

  • The training error increases as you increase the size of your dataset, because it becomes harder to fit a model that accounts for the increasing complexity/variability of your training set.
  • The test error decreases as you increase the size of your dataset, because the model is able to generalise better from a higher amount of information.

As you can see on the rightmost part of the plot, the two lines in the plot tend to reach and asymptote. Therefore, you eventually will reach a point in which increasing the size of your dataset will not have an impact on your trained model.

The distance between the test error and training error asymptotes is a representation of your model's overfitting. But more importantly, this plot is saying whether you need more data. Basically, if you represent test and training error for increasing larger subsets of your training data, and the lines do not seem to be reaching an asymptote, you should keep collecting more data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ In learning_curve function should I pass X_train, y_train: Only train subset or X, y: the entire dataset $\endgroup$
    – Rookie_123
    Commented Dec 3, 2018 at 15:45
  • $\begingroup$ This curve is built as a result of applying cross validation as you increase the number of samples. Therefore, you need the entire dataset. $\endgroup$
    – Pablo Suau
    Commented Dec 11, 2018 at 18:47
6
$\begingroup$
  1. Yes, the issue is certainly relevant, since your ability to fit the model will depend on the amount of data you have, but more importantly, it depends on the quality of the predictors.
  2. A 10-times rule might be a rule of thumb (and there are many others), but it really depends on the predictive utility of your features. E.g., the iris dataset is fairly small but easily solved, because the features yield good separation of the targets. Conversely, you could have 10 million examples and fail to fit if the features are weak.
Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! Can you also suggest me some papers or any material to read? $\endgroup$
    – DavideChicco.it
    Commented Jun 28, 2017 at 2:14
0
$\begingroup$

The "Rule of Ten" might work okay for some problems (e.g. linear, or a known functional form). This is NOT a general rule. All we need is one counter-example to show that this is not a general rule which holds. For proof (and a nice discussion), please see the paper by A. Siegel: https://www.sciencedirect.com/science/article/pii/S2772415821000110

In general, you need to know about your problem. You are guaranteed to be undersampled if you have less than 2^n (n = number of dimensions) samples. Even with this, you have the potential to miss features. Be careful when looking for general rules which have no mathematical backing!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.