1
$\begingroup$

I am training a machine learning model (i.e., a classifier) on a large dataset. I know that I can get the same results using less data (about 30%) but I would like to avoid the trial and error process to find the 'right' amount of data to retain from the dataset.

Of course I can create a script which automatically tried different thresholds but I was wondering if there is any principled way of doing this. It seems strange that nobody ever tried to create a proper solution since this seems to be a very common problem.

Some additional criteria:

  • I am using sub sampling on a stream of data so it would be better to find something that works in this setting
  • I would prefer to avoid training the classifier more than once since it takes some time
  • I appreciate theoretically justified approaches

Any suggestion or reference?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Every Machine Learning Problem is different, so there is no standard answer to your question. For the problem you're working on maybe a 70-30 train-test split would result in an optimal model which performs equally well on the test dataset, whereas for another problem may be that ratio just won't do any justice to the model. It's all about experimentation.

Basically, While training your model, you are basically trying to teach the model all relations, dependencies among attributes which can help the model create a clearer decision boundary between data points associated with your response variable.

Less training may not achieve the task as the model may not learn the underlying structure of the data and hence how the attributes are linked to the response variable, too much training data can have adverse affects too as then you may overfit. I would recommend that start from a 50-50 split record the model performance based on the metric you chose and then repeat the exercise for 60-40, 70-30 etc.

Improve this answer
$\endgroup$
5
  • $\begingroup$ The trial-and-error is what I am already doing. I would like to go beyond that. $\endgroup$
    – giz
    Apr 20, 2021 at 8:43
  • $\begingroup$ May give this a read : stackoverflow.com/questions/13610074/… $\endgroup$
    – user47
    May 18, 2021 at 16:28
  • $\begingroup$ In order to make minimum number of experiments you can train with 30% of data and 70% of data and compare results. If there is no really difference in terms of performance metrics you can use 30% of data. It is hard to estimate without experimenting, and making just two experiments is also not sufficient to get a reliable idea. $\endgroup$
    – tkarahan
    May 29, 2021 at 18:21
  • 1
    $\begingroup$ More training data typically leads to better generalization performance, not overfitting. $\endgroup$
    – Nuclear Hoagie
    Sep 27, 2021 at 18:11
  • $\begingroup$ The answers do not really answer the question. The user is asking weather one can sample the data as opposed to splitting the data into train test splits. Sampling means taking a chunk out of a large dataset and building a ML model and achieving same results, because building a model on the whole data would take up large amount of time and resources. $\endgroup$
    – spectre
    Jun 4, 2022 at 4:16
1
$\begingroup$

There are many rule based sampling techniques out there as opposed to just sampling randomly and hoping it works. The idea is to sample proportionately so that the model does not learn any biases or does not leave out minorities. Hope this helps!

PS: Some articles will prompt you to sign in after a few free articles. Just open them in icognito! ;)

Cheers!

Improve this answer
$\endgroup$
3
  • $\begingroup$ Sampling proportionally is definitely useful for small subsets, but when sampling tens of thousands of data points, this is often not a big concern. My problem is: given millions of data points, how many tens of thousands do I need to sample to approximate the original dataset well enough to train a model? $\endgroup$
    – giz
    Aug 9, 2022 at 10:15
  • $\begingroup$ @giz It depends on multiple factors. How many classes are we talking about? Is is an imbalanced or balanced classification problem? What algorithm are you using, Machine Learning or Deep Learning? I would suggest take an equal no of data points (how many depends on weather you are using a ML or DL model. If ML then you can get away with taking less data points, if DL you need more data) for each class and then train the classifier. $\endgroup$
    – spectre
    Aug 11, 2022 at 5:22
  • $\begingroup$ @giz Again it is an iterative process and you cannot say this is the best, smallest subsample based on 1 or 2 trial $\endgroup$
    – spectre
    Aug 11, 2022 at 5:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.