1
$\begingroup$

I train a binary random forest classifier on scikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold cross validation on the training data.

Is there any problem with this methodology?

For the max_depth parameter I get a really high value of 500 and that seems too high. Any advice?

The code is:

from __future__ import print_function  
import sklearn  
import sklearn.ensemble  
import sklearn.metrics  
from sklearn.datasets import fetch_20newsgroups  
from sklearn.grid_search import GridSearchCV

categories = ['sci.med', 'soc.religion.christian']  
newsgroups_train = fetch_20newsgroups(subset='train', categories=categories, 
remove=('headers', 'footers', 'quotes'))  
newsgroups_test = fetch_20newsgroups(subset='test', categories=categories, 
remove=('headers', 'footers', 'quotes'))  
class_names = ['medicine', 'christian']  

vectorizer = 
sklearn.feature_extraction.text.TfidfVectorizer(lowercase=False)  
train_vectors = vectorizer.fit_transform(newsgroups_train.data)  
test_vectors = vectorizer.transform(newsgroups_test.data)  

rf = sklearn.ensemble.RandomForestClassifier(max_features='sqrt')  

param_grid = {  
           "n_estimators" : [10, 100, 1000],  
           "max_depth" : [5, 100, 500],  
           "min_samples_leaf" : [1, 20, 40]}  

CV_rf = GridSearchCV(estimator=rf, param_grid=param_grid)  
CV_rf.fit(train_vectors, newsgroups_train.target)  
print(CV_rf.best_params_)
|improve this question|||||
$\endgroup$
2
$\begingroup$

Are you looking at the accuracy on your validation set, rather than your training set? (you should be). Are you making sure your gap between training and validation accuracy is low? (you should be!) Is there enough data to warrant 3-fold cross-validation, or should you do 10-fold (and use more data for training)?

In general, random search (where you sample randomly from the parameter space) will get you a good result faster than grid search. It has the added benefit that you can specify how many models you want to build, as each model's parameters are sampled independently and not constrained to cover an entire space like grid search.

|improve this answer|||||
$\endgroup$
1
$\begingroup$

It may be that your coarse scale is too large (though it is a good idea to begin). If for example 150 depth would be the best solution but 500 is better than 100 then it will give you 500 as a parameter result. Have you tried with shorter intervals between values for this parameter ? What does it give if you try differently?

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I ran param_grid = { "n_estimators" : [10, 20, 40, 80, 160, 320, 640, 1280], "max_depth" : [5, 10, 20, 40, 160, 320], "min_samples_leaf" : [1, 2, 4, 8, 16, 32, 64]} that took a while and the resuts were {'n_estimators': 160, 'max_depth': 320, 'min_samples_leaf': 2}. $\endgroup$ – Joe_base Nov 28 '17 at 19:40
  • $\begingroup$ When you keep the n_estimators and min_samples_leaf fixed (say, 160 and 2, respectively) and vary only the max_depth do you see any clear pattern in your metric or it is just jumping up and down? I suppose you don’t need to rerun for such analysis. $\endgroup$ – aivanov Nov 28 '17 at 21:12
  • $\begingroup$ It jumps with a tendency to going more up than down. On a equidistant scale it reached 400, more finegrained 420 and then 380. $\endgroup$ – Joe_base Nov 28 '17 at 22:49
0
$\begingroup$

max_depth of 500 cannot be right, if I do not mistake it implies about 2^500 leafs, which is not feasible.

You probably have some bugs in your code. Post your code if possible.

Also, are you sure you don’t mix up max_depth and number of trees?

|improve this answer|||||
$\endgroup$
  • $\begingroup$ i added the code. My idea was to search on a coarse scale and in a second step to make it more finegraines. {'n_estimators': 100, 'max_depth': 500, 'min_samples_leaf': 1} was the first result and after some iterations I arrived at 'n_estimators': 110, 'max_depth': 510, 'min_samples_leaf': 2} $\endgroup$ – Joe_base Nov 28 '17 at 17:07
0
$\begingroup$

500 can be right. It does not imply 2^500 leafs, some leafs can stop earlier. How many observations do you have?

In random forest you could use the out-of-bag predictions for tuning. That would make your tuning algorithm faster.

Max_depth = 500 does not have to be too much. The default of random forest in R is to have the maximum depth of the trees, so that is ok. You should validate your final parameter settings via cross-validation (you then have a nested cross-validation), then you could see if there was some problem in the tuning process.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.