0
$\begingroup$

As the huge title says I'm trying to use GridSearchCV to find the best parameters for a Random Forest Regressor and I'm measuring my results with mse.

Inputs_Treino = dataset.iloc[:253,1:4].values
Outputs_Treino = dataset.iloc[:253,-1].values
Inputs_Teste = dataset.iloc[254:,1:4].values
Outputs_Teste = dataset.iloc[254:,-1].values

estimator = RandomForestRegressor()
para_grids = {
            "n_estimators" : [10,50,100],
            "max_features" : ["auto", "log2", "sqrt"],
            "bootstrap"    : [True, False]
        }


grid = GridSearchCV(estimator, para_grids, scoring = 'mean_squared_error')
grid.fit(Inputs_Treino, Outputs_Treino)
forest = grid.best_estimator_

reg_prediction=forest.predict(Inputs_Teste)

print (grid.best_score_, grid.best_params_)

mse = mean_absolute_error(Outputs_Teste, reg_prediction)

This is the gist of the code (nothing too complex I know, just getting started with it all)

When I print the result of grid.best_estimator_ I get this

RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_split=1e-07, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
           verbose=0, warm_start=False)

The problem is if I try to create a regressor with these parameters (without using grid search at all) and train it the same way I get a waaaay bigger MSE on the testing set (5.483837301587303 vs 43.801520165079467)

Inputs_Treino = dataset.iloc[:253,1:4].values
Outputs_Treino = dataset.iloc[:253,-1].values
Inputs_Teste = dataset.iloc[254:,1:4].values
Outputs_Teste = dataset.iloc[254:,-1].values

regressor = RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_split=1e-07, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
           verbose=0, warm_start=False)

regressor.fit(Inputs_Treino,Outputs_Treino)

#fazer as predictions
Teste_Prediction = regressor.predict(Inputs_Teste);

mse = mean_squared_error(Outputs_Teste, Teste_Prediction);

Does this have to do with the cross validation GridSearchCV performs ? What am I missing here ?

Improve this question
$\endgroup$
3
  • $\begingroup$ n_estimators=10, VS n_estimators=100, - maybe not the reason, but... $\endgroup$
    – f.g.
    Commented Oct 15, 2018 at 21:16
  • $\begingroup$ @f.g. oh that's a mistake on my part but the result is the same. $\endgroup$
    – Gabriel Silva
    Commented Oct 15, 2018 at 21:31
  • $\begingroup$ Also, mean_absolute_error vs mean_squared_error. That seems reasonably likely to explain at least a large part of the difference. $\endgroup$
    – Ben Reiniger ♦
    Commented Apr 15, 2019 at 19:23

1 Answer 1

Reset to default
1
$\begingroup$

RandomForest has randomness in the algorithm. First, when it bootstrap samples the data for each tree. Second, when it chooses random subsamples of features for each split.

To reproduce results across runs you should set the random_state parameter. For example:

estimator = RandomForestRegressor(random_state=420)
Improve this answer
$\endgroup$
1
  • $\begingroup$ This is an excellent point, and seems to be the right answer to the title question, but is such a large difference expected? $\endgroup$
    – Ben Reiniger ♦
    Commented Apr 15, 2019 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.