0
$\begingroup$

As the huge title says I'm trying to use GridSearchCV to find the best parameters for a Random Forest Regressor and I'm measuring my results with mse.

Inputs_Treino = dataset.iloc[:253,1:4].values
Outputs_Treino = dataset.iloc[:253,-1].values
Inputs_Teste = dataset.iloc[254:,1:4].values
Outputs_Teste = dataset.iloc[254:,-1].values

estimator = RandomForestRegressor()
para_grids = {
            "n_estimators" : [10,50,100],
            "max_features" : ["auto", "log2", "sqrt"],
            "bootstrap"    : [True, False]
        }


grid = GridSearchCV(estimator, para_grids, scoring = 'mean_squared_error')
grid.fit(Inputs_Treino, Outputs_Treino)
forest = grid.best_estimator_

reg_prediction=forest.predict(Inputs_Teste)

print (grid.best_score_, grid.best_params_)

mse = mean_absolute_error(Outputs_Teste, reg_prediction)

This is the gist of the code (nothing too complex I know, just getting started with it all)

When I print the result of grid.best_estimator_ I get this

RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_split=1e-07, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
           verbose=0, warm_start=False)

The problem is if I try to create a regressor with these parameters (without using grid search at all) and train it the same way I get a waaaay bigger MSE on the testing set (5.483837301587303 vs 43.801520165079467)

Inputs_Treino = dataset.iloc[:253,1:4].values
Outputs_Treino = dataset.iloc[:253,-1].values
Inputs_Teste = dataset.iloc[254:,1:4].values
Outputs_Teste = dataset.iloc[254:,-1].values

regressor = RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_split=1e-07, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
           verbose=0, warm_start=False)

regressor.fit(Inputs_Treino,Outputs_Treino)

#fazer as predictions
Teste_Prediction = regressor.predict(Inputs_Teste);

mse = mean_squared_error(Outputs_Teste, Teste_Prediction);

Does this have to do with the cross validation GridSearchCV performs ? What am I missing here ?

$\endgroup$
  • $\begingroup$ n_estimators=10, VS n_estimators=100, - maybe not the reason, but... $\endgroup$ – f.g. Oct 15 '18 at 21:16
  • $\begingroup$ @f.g. oh that's a mistake on my part but the result is the same. $\endgroup$ – Gabriel Silva Oct 15 '18 at 21:31
  • $\begingroup$ Also, mean_absolute_error vs mean_squared_error. That seems reasonably likely to explain at least a large part of the difference. $\endgroup$ – Ben Reiniger Apr 15 at 19:23
1
$\begingroup$

RandomForest has randomness in the algorithm. First, when it bootstrap samples the data for each tree. Second, when it chooses random subsamples of features for each split.

To reproduce results across runs you should set the random_state parameter. For example:

estimator = RandomForestRegressor(random_state=420)
$\endgroup$
  • $\begingroup$ This is an excellent point, and seems to be the right answer to the title question, but is such a large difference expected? $\endgroup$ – Ben Reiniger Apr 15 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.