1
$\begingroup$

I am using RandomForestRegressor (scikit-learn python package). I am looking for the best values for hyperparameters n_estimators and min_samples_split for fitting my regressor on a train dataset ( X_train, y_train ) :

  param_grid = { 'n_estimators' : range( 10 , 201 ) ,
                 'min_samples_split' : range( 2 , 11 ) }


  rfr = RandomForestRegressor()

  best_score = 0
  best_param = {}

  for param in ParameterGrid( param_grid ):
      rfr.set_params( **g )
      rfr.fit( X_train , y_train )
      score = rfr.score( X_train , y_train )

  if score > best_score :
      best_score = score
      best_param = param

  print( 'best_score : {0}'.format( best_score ) )
  print( 'best_parameters : {0}'.format( best_param ) )

I am not using cross validation on purpose.

The best hyper parameters are in the best_param variable , the corresponding score in the best_score variable.

After that, I set the hyper parameters of my regressor using the best ones :

rfr.set_params( **best_param )

I fit the regressor on my train dataset:

rfr.fit( X_train , y_train )

Finally just to check I obtain the score of the regressor with the best hyper parameters:

rfr.score( X_train , y_train )

The score is not the same as the one calculated in the grid : 0.806 out of grid versus 0.963 in the grid.

I do not understand why. The hyper parameters are the best ones, and the dataset used to calculate these best parameters is the same. Any hints?

$\endgroup$

2 Answers 2

3
$\begingroup$

I have found the answer to my question : there is randomness in RandomForestRegressor().

Using the random_state hyper parameter fix the problem :

rfr = RandomForestRegressor( random_state = 123 )

What is surprising is the difference of score obtained from different values of the random_state hyper parameter. It means my best hyper parameters n_estimators and min_samples_split are strongly depend on the random_state hyper parameter.

I guess it is something I will have to deal with.

$\endgroup$
1
$\begingroup$

Without cross validation you are effectively selecting the best hyperparameters for just one set of training data so its overfitting. However, because of the random_state parameter not being set, you are basically using different samples of your training data each time.

See this cross-validated question: How bad is hyperparameter tuning outside cross validation? Maybe you should try it with CV and see if that still happens.

$\endgroup$
2
  • $\begingroup$ Thanks for the link about hyperparameter tuning outside cross validation. I am not using cross validation because my data is some kind of time serie. I have to check this link that seems promising : scikit-learn.org/stable/modules/generated/… $\endgroup$ Aug 14, 2019 at 5:31
  • $\begingroup$ Oh cool. I don't know much about cv for time series. Could maybe be an interesting question to post here! $\endgroup$ Aug 14, 2019 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.