0
$\begingroup$

I often work with very large datasets where it would be impractical to check all relevant combinations of hyperparameters when constructing a machine learning model. I'm considering randomly sampling my dataset and then performing hyperparameter tuning using the sample. Then, I would train/test the model using the full dataset with the chosen hyperparameters.

What are the disadvantages of this approach?

$\endgroup$
0
$\begingroup$

One of the good practices is to create a split in the dataset for each tuning/ training step of your pipeline. Since you have large datasets, you should have enough data to split the original dataset into multiple subsets and still have a relevant number of rows for each step. As such, as an example, you can divide your dataset in 60% training, 20% hyperparameter tuning and 20% for the test.

It is important to avoid optimizing the hyperparameters with the same data you train on because this can lead to overfitting both tuning steps of your model to the same source of data.

Also, be careful on how you sample the original datasource. When dealing with highly skewed categorical features, random sampling can lead to categories in the test set which are not observed during training, which can cause some models to break. Also, numerical features should have a similar distribution between the training and the test set.

| improve this answer | |
$\endgroup$
0
$\begingroup$

You could perhaps use nested cross-validation, where you divide the data into N folds, and then you divide each fold further into a training and test set and use them to find the best hyperparameters for that fold, and so on. It is best explained here:

https://stats.stackexchange.com/a/45739

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.