0
$\begingroup$

If I used a subset of the entire available training data for model tuning and hyperparamater selection, should I fit the final model to the subset training dataset or the entire available training data? For example, if I have 1M samples available and I took a 100K random samples as a test holdout and 200K random samples as a training dataset for model tuning, should the tuned hyperparamaters used to fit the final model on the 1) 200K training dataset, or 2) 900K available data (excluding test holdout)? In other words, can the hyperparameters by generalized for the entire population?

I am assuming that both the holdout and training datasets are selected randomly and follow the class distribution in the original data.

$\endgroup$
2
  • $\begingroup$ First question: why are you only training on 200k records? What are you doing with the other 700k? $\endgroup$
    – astel
    Jun 24 '20 at 16:14
  • $\begingroup$ For speed and due to memory and hardware limitation. $\endgroup$ Jun 24 '20 at 17:50
1
$\begingroup$

The general machine learning process is this:

Split your data into two parts, training and test. So in your example I would take 100k for test and 900k for training (don't know why you say only take 200k in your question but I digress). With the 900k training we perform hyper-parameter tuning. This can be done by splitting training into training and validation say 800k/100k or better yet we could do this using k-fold cross validation.

Once you have chosen the optimal hyper-parameters in this manner you evaluate their performance on the test set. The whole point of this process is simply to evaluate the algorithms performance, and from that select and algorithm. That is the only reason for a train/validation/test split. (As a note, this process can be even further improved by using something called nested cross-validation but I will not go into the details).

After you have selected your algorithm and determined its performance (error rate), you take your whole data set (1 million records) and perform hyper-parameter selection on that, either using a single split or by k-fold cross-validation. You no longer need the test set because you have already determined the models error rate in the previous step.

Once you have selected the best hyper-parameters in the previous step you apply them to the entire data set (the 1 million records) and build the model.

$\endgroup$
4
  • $\begingroup$ Thank you for taking the time to answer the question. I am well aware of the model building process you mentioned, in addition to nested CV. If I understand your answer currently, then you are recommend two rounds of hyperparametr tuning. is that correct? I am not clear why do you need to tune on 800K samples and then on 1M samples again. What if I am unable to use the entire 900K samples for tuning due to cpu and memory capability, can I used a subset of this data (i.e., 200K samples) for tuning and then fit the final model on the entire data (i.e., 1M samples)? $\endgroup$ Jun 24 '20 at 18:03
  • 1
    $\begingroup$ You only tune your hyper parameters again because tuning on more data should give you better results. Essentially, if you’ve already determine your error rate, there is no longer a need for a test set so you should make use of it to improved your model. However, if you are stuck using only 200k records then no, there is no need to run the hyper-parameter search again, once will do. But, I would also add that if your training data is limited to 200k you’re test set should be 800k not 100k. Make use of all of your data. $\endgroup$
    – astel
    Jun 24 '20 at 18:55
  • $\begingroup$ Indeed using all the data makes sense. Am I also correct to assume no issue with the generalization of the hyperparameters from 200K to 1M as long as the 200K are indeed a random sample of the entire population? $\endgroup$ Jun 24 '20 at 19:09
  • 1
    $\begingroup$ Yes, so long as your data is homogenous enough such that the 200k is a representative sample of that 1 million. $\endgroup$
    – astel
    Jun 24 '20 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.