ScikitLearn - RandomForestRegressor score different in and out of grid search

Question

I am using RandomForestRegressor (scikit-learn python package). I am looking for the best values for hyperparameters n_estimators and min_samples_split for fitting my regressor on a train dataset ( X_train, y_train ) :

  param_grid = { 'n_estimators' : range( 10 , 201 ) ,
                 'min_samples_split' : range( 2 , 11 ) }


  rfr = RandomForestRegressor()

  best_score = 0
  best_param = {}

  for param in ParameterGrid( param_grid ):
      rfr.set_params( **g )
      rfr.fit( X_train , y_train )
      score = rfr.score( X_train , y_train )

  if score > best_score :
      best_score = score
      best_param = param

  print( 'best_score : {0}'.format( best_score ) )
  print( 'best_parameters : {0}'.format( best_param ) )

I am not using cross validation on purpose.

The best hyper parameters are in the best_param variable , the corresponding score in the best_score variable.

After that, I set the hyper parameters of my regressor using the best ones :

rfr.set_params( **best_param )

I fit the regressor on my train dataset:

rfr.fit( X_train , y_train )

Finally just to check I obtain the score of the regressor with the best hyper parameters:

rfr.score( X_train , y_train )

The score is not the same as the one calculated in the grid : 0.806 out of grid versus 0.963 in the grid.

I do not understand why. The hyper parameters are the best ones, and the dataset used to calculate these best parameters is the same. Any hints?

Answer 1

I have found the answer to my question : there is randomness in RandomForestRegressor().

Using the random_state hyper parameter fix the problem :

rfr = RandomForestRegressor( random_state = 123 )

What is surprising is the difference of score obtained from different values of the random_state hyper parameter. It means my best hyper parameters n_estimators and min_samples_split are strongly depend on the random_state hyper parameter.

I guess it is something I will have to deal with.

Answer 2

Without cross validation you are effectively selecting the best hyperparameters for just one set of training data so its overfitting. However, because of the random_state parameter not being set, you are basically using different samples of your training data each time.

See this cross-validated question: How bad is hyperparameter tuning outside cross validation? Maybe you should try it with CV and see if that still happens.