0
$\begingroup$

Can Gridsearchcv params perform worst than default RF?

RF with default values performs rmse_train=4886,r^2_train=0.84, rmse_test=11008,r^2_test=0.22. RF after GridSearchCV tuning performs worst on train set (rmse_train=9104,r^2_train=0.45, rmse_test=11091,r^2_test=0.21). This is the code (my first ML algorithm implementation)

#data
features = pd.read_csv("dati_nn.csv")
labels = np.array(features['Cost_damage']) #regression problem
features = features.drop('Cost_damage', axis=1)
features = np.array(features)
train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size=0.2, random_state = 123)

#RF default
rf = RandomForestRegressor(random_state = 123)
rf.fit(train_features, train_labels)
pred_train = rf.predict(train_features)
predictions = rf.predict(test_features)
# compute rmse and r_score
rmse_train = np.sqrt(metrics.mean_squared_error(train_labels, pred_train))
rmse_test = np.sqrt(metrics.mean_squared_error(test_labels, predictions))
r2_train = r2_score(train_labels, pred_train)
r2_test = r2_score(test_labels, predictions)

#GridSearch
param_grid = {
    'bootstrap': [True], 
    'max_depth': [20, 30], 
    'max_features': ['auto', 'sqrt'], 
    'min_samples_leaf': [1, 2, 3], 
    'min_samples_split': [5, 8, 10, 12], 
    'n_estimators': [500, 600, 700, 800, 1000] 
}
rf = RandomForestRegressor(random_state = 123)
grid_search = GridSearchCV(estimator = rf, 
                           param_grid = param_grid, 
                           scoring = 'neg_mean_squared_error', 
                           cv = 10, 
                           n_jobs = -1,
                           verbose = 2 
                          )
grid_search.fit(train_features, train_labels)

best_grid = grid_search.best_estimator_
pred_train = best_grid.predict(train_features)
predictions = best_grid.predict(test_features)
#rmse and r2_score as above

In my opinion, the code is correct but I don't understand why rmse grows. I searched for many days on the internet to find an answer...no success. I tried to use RandomizedSearchCV, same problem. Any suggest? Is my workflow correct?

Thanks in advance!!

$\endgroup$

1 Answer 1

0
$\begingroup$

These are different for a valid reason
The standalone model is overfitting. You can see that with r2score of 84 Vs 22.

The reason for that is, the standalone model goes to full depth and hence overfit the train data and badly fit the test data.

max_depthint, default=None
The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.

While with CV, you have parm for max_depth. Hence bad but Truth.

$\endgroup$
1
  • $\begingroup$ Oh, it makes sense...with r2=0.84 I didn't think about overfitting. So, the steps are correct but I've to play with hyperparms. Can be useful run RandomizedSearchCV before Gridsearch to limit the hyperparms range? like in this post link $\endgroup$
    – simo954
    Commented May 5, 2021 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.