I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process to find the 'right' amount of data to retain from the dataset.

Of course I can create a script which automatically tried different thresholds but I was wondering if there is any principled way of doing this. It seems strange that nobody ever tried to create a proper solution since this seems to be a very common problem.

Some additional criteria:

  • I am using sub sampling on a stream of data so it would be better to find something that works in this setting
  • I would prefer to avoid training the classifier more than once since it takes some time
  • I appreciate theoretically justified approaches

Any suggestion or reference?


Every Machine Learning Problem is different, so there is no standard answer to your question. For the problem you're working on maybe a 70-30 train-test split would result in an optimal model which performs equally well on the test dataset, whereas for another problem may be that ratio just won't do any justice to the model. It's all about experimentation.

Basically, While training your model, you are basically trying to teach the model all relations, dependencies among attributes which can help the model create a clearer decision boundary between data points associated with your response variable.

Less training may not achieve the task as the model may not learn the underlying structure of the data and hence how the attributes are linked to the response variable, too much training data can have adverse affects too as then you may overfit. I would recommend that start from a 50-50 split record the model performance based on the metric you chose and then repeat the exercise for 60-40, 70-30 etc.

  • $\begingroup$ The trial-and-error is what I am already doing. I would like to go beyond that. $\endgroup$
    – giz
    Apr 20 at 8:43
  • $\begingroup$ May give this a read : stackoverflow.com/questions/13610074/… $\endgroup$
    – user47
    May 18 at 16:28
  • $\begingroup$ In order to make minimum number of experiments you can train with 30% of data and 70% of data and compare results. If there is no really difference in terms of performance metrics you can use 30% of data. It is hard to estimate without experimenting, and making just two experiments is also not sufficient to get a reliable idea. $\endgroup$
    – tkarahan
    May 29 at 18:21
  • $\begingroup$ More training data typically leads to better generalization performance, not overfitting. $\endgroup$ Sep 27 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.