Is it a good idea to tune the number of folds for cross validation when tuning hyperparameters of RF

Question

I'm new to data science. I'm trying to get the best model for Random Forest. Unfortunately, I'm not sure if my idea can produce a good generalized model.

1) I have split data to TrainingSet (70%) and TestSet (30%)

2) Then randomly selected hyperparameters for RandomForest and a number of folds for CrossValidation between (2-15)

3) Then I fetch the TraingSet data to the RandomForest learner

4) Then do CrossValidation of the model - from CrossValidation I'm getting array with predictions

5) Measure Accuracy of prediction from CrossValidation against targets from the TrainSet

6) Repeat all steps and try minimize the AccuracyError

Is this a good way to get best generalized model?

Do I need to split data into TrainSet and TestSet?

OR I should I search for the optimal hyperparameters and number of folds with all data? I have feeling I don't need to split data when using k-fold CrossValidation during Hyperparameters tunning.

Answer 1

Do I need to split data to TrainSet and TestSet?

It depends:

• It is acceptable to cross-validate on the whole dataset to obtain the performance of a particular model, since in this case the model is always tested on unseen data.
• In your case, the selection of the best generalized model/hyperparameters is part of the training stage, so yes it makes sense to cross-validate only on your training set and then measure the performance of your final model on some fresh data. This is because the selection of the best hyperparameters (especially from a large set of possibilities) can still be partly due to chance despite CV (especially if the dataset is small), so testing on unseen data avoids the risk of overestimating the performance.

Then randomly select hyperparameters for RandomForest and a number of folds for CrossValidation between (2-15)

Usually the number of folds k for k-fold CV is not really part of the hyper-parameters, so it's a quite unusual to do this (as far as I know). The risk is that your final model might be selected partly because a particular k happens to produce a higher performance on the training set. To see this intuitively: a larger training set is more likely to produce a better model, and a higher k makes CV use a larger training set for every fold. So a higher k is likely to artificially improve the performance, just because the training set is larger. This is why it's safer to use the same k for comparing different hyper-parameters, possibly repeating the whole process for different values of k. Also for a particular value of k you can randomly reshuffle the partitions to minimize the effect of chance in the CV partitioning.

Answer 2

Sounds okay. However, I assume you make a prediction for each fold and average them to test the accuracy?

There are basically three ways to tune hyperparameters. Random search (as you do), gridsearch (search over a range of values), and Bayesian optimization. The latter can be much more efficient.

Here is a good resource for BO: http://krasserm.github.io/2018/03/21/bayesian-optimization/