# ScikitLearn - RandomForestRegressor score different in and out of grid search

I am using RandomForestRegressor (scikit-learn python package). I am looking for the best values for hyperparameters n_estimators and min_samples_split for fitting my regressor on a train dataset ( X_train, y_train ) :

  param_grid = { 'n_estimators' : range( 10 , 201 ) ,
'min_samples_split' : range( 2 , 11 ) }

rfr = RandomForestRegressor()

best_score = 0
best_param = {}

for param in ParameterGrid( param_grid ):
rfr.set_params( **g )
rfr.fit( X_train , y_train )
score = rfr.score( X_train , y_train )

if score > best_score :
best_score = score
best_param = param

print( 'best_score : {0}'.format( best_score ) )
print( 'best_parameters : {0}'.format( best_param ) )


I am not using cross validation on purpose.

The best hyper parameters are in the best_param variable , the corresponding score in the best_score variable.

After that, I set the hyper parameters of my regressor using the best ones :

rfr.set_params( **best_param )


I fit the regressor on my train dataset:

rfr.fit( X_train , y_train )


Finally just to check I obtain the score of the regressor with the best hyper parameters:

rfr.score( X_train , y_train )


The score is not the same as the one calculated in the grid : 0.806 out of grid versus 0.963 in the grid.

I do not understand why. The hyper parameters are the best ones, and the dataset used to calculate these best parameters is the same. Any hints?

I have found the answer to my question : there is randomness in RandomForestRegressor().

Using the random_state hyper parameter fix the problem :

rfr = RandomForestRegressor( random_state = 123 )


What is surprising is the difference of score obtained from different values of the random_state hyper parameter. It means my best hyper parameters n_estimators and min_samples_split are strongly depend on the random_state hyper parameter.

I guess it is something I will have to deal with.

Without cross validation you are effectively selecting the best hyperparameters for just one set of training data so its overfitting. However, because of the random_state parameter not being set, you are basically using different samples of your training data each time.

See this cross-validated question: How bad is hyperparameter tuning outside cross validation? Maybe you should try it with CV and see if that still happens.

• Thanks for the link about hyperparameter tuning outside cross validation. I am not using cross validation because my data is some kind of time serie. I have to check this link that seems promising : scikit-learn.org/stable/modules/generated/… – Fabrice BOUCHAREL Aug 14 '19 at 5:31
• Oh cool. I don't know much about cv for time series. Could maybe be an interesting question to post here! – fractalnature Aug 14 '19 at 14:56