0
$\begingroup$

I'm working with a supervised learning problem and trying to predict a binary label and using a Random Forest to do so.

I'm trying to tune my hyper-parameters to give me a best model based on my data. I can do this with GridSearchCV(), but is this correct to do with a random forest? If I'm using GridSearchCV(), the training set and testing set change with each fold. From my understanding we can we set oob_true = True in RandomForestClassifier(), we are already evaluating on the out-of-bag samples (so CV is kind of already built in RF).

What is the convention to hyper-parameter tune with Random Forest to get the best OOB score in sklearn? Can I just loop through a set of parameters and fit on the same training and testing set? Can I use GridSearchCV(), or does that make no sense with RF?

Improve this question
$\endgroup$
0
$\begingroup$

You can definitely use GridSearchCV with Random Forest. In fact you should use GridSearchCV to find the best parameters that will make your oob_score very high.

Some parameters to tune are:

n_estimators: Number of tree your random forest should have. The more n_estimators the less overfitting. You should try from 100 to 5000 range.

max_depth: max_depth of each tree. You should specify certain max_depth so that your model don't memorise train examples.

min_sample_split: the minimum number of samples to have before splitting into new nodes.

and many more...

But these are the main hyperparameters we tune to get a forest that works well.And to get these hyperparams you should use GridSeachCV.

Improve this answer
$\endgroup$
10
  • $\begingroup$ The main advantage of Random Forest is that it is not prone to overfitting!!! $\endgroup$ – nimar Jun 19 '20 at 19:18
  • 2
    $\begingroup$ If not regularized properly, Random Forest also overfits. $\endgroup$ – SrJ Jun 19 '20 at 19:21
  • $\begingroup$ That could be true about the decision tree, not RF. In general, ensembling methods are less prone to overfitting. $\endgroup$ – nimar Jun 19 '20 at 19:52
  • $\begingroup$ If I use GridSearchCV won't I be essentially performing CV twice? Since Random Forest already peforms bagging right? $\endgroup$ – Eisen Jun 19 '20 at 19:52
  • $\begingroup$ In a sense yes. But GridSearchCV is used to find the optimal hyperparameters. So if you want to search for the best hyperparams you should go for GridSearchCV. If you are only doing cross validation, you may not use GridSearchCV $\endgroup$ – SrJ Jun 19 '20 at 19:58
0
$\begingroup$

You can very well use the GridSearchCV to fine tune RandomForest.

I do not understand what you mean by "If I'm using GridSearchCV(), the training set and testing set change with each fold."

Generally we apply GridSearchCV on the test_data set after we do the train test split. The Crossvalidation splits the training data into multiple train and test split based on the Kfold value that you give. For example if k value is 10 then we will have the training data splitted into 10 folds where 1 will be used for testing and 9 together used for training. This happens until all the 10 folds are used for testing so you will get 10 accuracy score. In addition to this in gridsearchcv we pass a set of hyper parameters based on the model that we are using. This helps is finding the best hyperparameters for the model to get the best accuracy score and also to avoid overfitting.

On the other hand oob is some unseen data by the random forest model.

Let me know if you need more information in detail.

Hope this is helpful

Improve this answer
$\endgroup$
0
$\begingroup$

I can do this with GridSearchCV(), but is this correct to do with a random forest?

Yes, this is perfectly valid. It ignores the oob-score feature of random forests, but that isn't necessarily a bad thing. See e.g. https://stats.stackexchange.com/a/462720/232706

What is the convention to hyper-parameter tune with Random Forest to get the best OOB score in sklearn? Can I just loop through a set of parameters and fit on the same training and testing set?

I believe this would be the standard way of tuning using oob score, except that there is no testing set in this case. (You'll probably want a test set for future performance estimation of the final selected model though: selecting hyperparameters based on those oob scores means they are no longer unbiased estimates of future performance, just as in k-fold cross-validation! Your hyperparameter-candidate models shouldn't see that test set.)

To take advantage of the various conveniences of the hyperparameter searches in sklearn (parallelization, saved results, refitted best model, etc.), you can hack it as in my answer to another question:
https://datascience.stackexchange.com/a/66238/55122
You can't directly use oob score in a GridSearchCV because that's coded to apply your scoring function to the test fold in each split.

Improve this answer
$\endgroup$
4
  • $\begingroup$ What do you mean by " except that there is no testing set in this case"? My understanding was I can split the full data set to a training and testing set. Then loop through a set of parameters for the training set with the goal of getting the optimal OOB score. I can then evaluate and get a score of OOB on the testing set? Also, thanks for these links, they make things alot easier to understand. $\endgroup$ – Eisen Jun 21 '20 at 4:12
  • $\begingroup$ @Eisen How do you mean to "evaluate and get a score of OOB on the testing set"? OOB is explicitly a score from the training set, by subsets of the trees. You can leave your test set for a final model evaluation, but I wouldn't call that OOB. $\endgroup$ – Ben Reiniger Jun 21 '20 at 15:31
  • $\begingroup$ I'm looking at your notebook (github.com/bmreiniger/datascience.stackexchange/blob/master/…). You have PredefinedSplit (i'm not really sure what this means, im assuming the training and testing is always the same?), but you end up getting only one split. Which means training/testing set is always the same?. From there you print out the cv results from what you trained on. Because random forest uses bagging and already is evaluated on the OOB set, the OOB score from the grid search should be good for evaluation already, right? $\endgroup$ – Eisen Jun 21 '20 at 19:38
  • $\begingroup$ @Eisen, yes, PredefinedSplit is there to allow a single train/test split as the cv parameter of a search. And the point in that question is to use the training oob score in the search (which is what will be stored in the cv_results_, and used to refit a best model, etc.), not the usual k-fold cv scoring. $\endgroup$ – Ben Reiniger Jun 22 '20 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.