Parameter Tuning by Cross Validation for Random Forest

Question

I train a binary random forest classifier on scikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold cross validation on the training data.

Is there any problem with this methodology?

For the max_depth parameter I get a really high value of 500 and that seems too high. Any advice?

The code is:

from __future__ import print_function  
import sklearn  
import sklearn.ensemble  
import sklearn.metrics  
from sklearn.datasets import fetch_20newsgroups  
from sklearn.grid_search import GridSearchCV

categories = ['sci.med', 'soc.religion.christian']  
newsgroups_train = fetch_20newsgroups(subset='train', categories=categories, 
remove=('headers', 'footers', 'quotes'))  
newsgroups_test = fetch_20newsgroups(subset='test', categories=categories, 
remove=('headers', 'footers', 'quotes'))  
class_names = ['medicine', 'christian']  

vectorizer = 
sklearn.feature_extraction.text.TfidfVectorizer(lowercase=False)  
train_vectors = vectorizer.fit_transform(newsgroups_train.data)  
test_vectors = vectorizer.transform(newsgroups_test.data)  

rf = sklearn.ensemble.RandomForestClassifier(max_features='sqrt')  

param_grid = {  
           "n_estimators" : [10, 100, 1000],  
           "max_depth" : [5, 100, 500],  
           "min_samples_leaf" : [1, 20, 40]}  

CV_rf = GridSearchCV(estimator=rf, param_grid=param_grid)  
CV_rf.fit(train_vectors, newsgroups_train.target)  
print(CV_rf.best_params_)  

Answer 1

Are you looking at the accuracy on your validation set, rather than your training set? (you should be). Are you making sure your gap between training and validation accuracy is low? (you should be!) Is there enough data to warrant 3-fold cross-validation, or should you do 10-fold (and use more data for training)?

In general, random search (where you sample randomly from the parameter space) will get you a good result faster than grid search. It has the added benefit that you can specify how many models you want to build, as each model's parameters are sampled independently and not constrained to cover an entire space like grid search.

Answer 2

It may be that your coarse scale is too large (though it is a good idea to begin). If for example 150 depth would be the best solution but 500 is better than 100 then it will give you 500 as a parameter result. Have you tried with shorter intervals between values for this parameter ? What does it give if you try differently?

Answer 3

max_depth of 500 cannot be right, if I do not mistake it implies about 2^500 leafs, which is not feasible.

You probably have some bugs in your code. Post your code if possible.

Also, are you sure you don’t mix up max_depth and number of trees?

Answer 4

500 can be right. It does not imply 2^500 leafs, some leafs can stop earlier. How many observations do you have?

In random forest you could use the out-of-bag predictions for tuning. That would make your tuning algorithm faster.

Max_depth = 500 does not have to be too much. The default of random forest in R is to have the maximum depth of the trees, so that is ok. You should validate your final parameter settings via cross-validation (you then have a nested cross-validation), then you could see if there was some problem in the tuning process.

Answer 5

Running this now, and updating to import GridSearchCV from model_selection, the code is:

import sklearn  
import sklearn.ensemble  
import sklearn.metrics  
from sklearn.datasets import fetch_20newsgroups  
# from sklearn.grid_search import GridSearchCV
from sklearn.model_selection import GridSearchCV

categories = ['sci.med', 'soc.religion.christian']  
newsgroups_train = fetch_20newsgroups(subset='train', categories=categories, 
remove=('headers', 'footers', 'quotes'))  
newsgroups_test = fetch_20newsgroups(subset='test', categories=categories, 
remove=('headers', 'footers', 'quotes'))  
class_names = ['medicine', 'christian']  

vectorizer = sklearn.feature_extraction.text.TfidfVectorizer(lowercase=False)  
train_vectors = vectorizer.fit_transform(newsgroups_train.data)  
test_vectors = vectorizer.transform(newsgroups_test.data)  

rf = sklearn.ensemble.RandomForestClassifier(max_features='sqrt')  

param_grid = {  
           "n_estimators" : [10, 100, 1000],  
           "max_depth" : [5, 100, 500],  
           "min_samples_leaf" : [1, 20, 40]}  

CV_rf = GridSearchCV(estimator=rf, param_grid=param_grid)  
CV_rf.fit(train_vectors, newsgroups_train.target)  
print(CV_rf.best_params_)  

Which outputs:

{'max_depth': 100, 'min_samples_leaf': 1, 'n_estimators': 1000}