Search for hyperparameters whith different features using Random Forest

Question

I have a dataset in which I would like to perform a classification model, so I have decided to use Random Forest. The number of features that I have is approximately 200 and I would like to test which set of features gives me a better result. For the sake of experimentation, I do not want to use any feature importance method.

So what I have done is to split my features, without replacement, in groups of 20, 40,60,80 and 100 features. I am using GridSearch for tuning the hyperparameters of my RF. The question that I have is if I want to compare the accuracy of each one of the models (the one with 20 features, with 40, and so on), it would be fair to apply GridSearch with each subset of the features. Or should I only perform Gridsearch once, let's say with the model of 20 features, and then use the same hyperparameters with the set of 40 features, 60 features and so on.

Any help?

Thanks

Answer 1

By performing GridSearch I understand you want to say searching for the best hyperparameters.

For sake of simplicity, let say that you want to fit a linear regression with a penalty (lasso/ridge) with 1 feature and with 100 features. The hyperparameter that you are looking for is the $\lambda$ penalty.

It is easy to see that with 1 feature your model might need some parameter, it could be that the feature has a perfect distribution and it doesn´t need any penalty at all. But when we go for 100 features there is some noise and you might need penalties to make sure that your model is generalizing well. So $\lambda$ will be different

With this example, my idea is to show that hyper-parametrization is a specific problem for each task.

For Random Forest, it would be the same. Different features will require different parameters so yes, you have to do a GridSearchCV with every subset of features if you want to achieve optimality