0
$\begingroup$

I first construct a base model (using default parameters) and obtain MAE.

# BASELINE MODEL
rfr_pipe.fit(train_x, train_y)
base_rfr_pred = rfr_pipe.predict(test_x)
base_rfr_mae = mean_absolute_error(test_y, base_rfr_pred)

MAE = 2.188

Then I perform GridSearchCV to get best parameters and get the average MAE.

# RFR GRIDSEARCHCV
rfr_param = {'rfr_model__n_estimators' : [10, 100, 500, 1000], 
         'rfr_model__max_depth' : [None, 5, 10, 15, 20],
         'rfr_model__min_samples_leaf' : [10, 100, 500, 1000],
         'rfr_model__max_features' : ['auto', 'sqrt', 'log2']}
rfr_grid = GridSearchCV(estimator = rfr_pipe, param_grid = rfr_param, n_jobs = -1,
                    cv = 5, scoring = 'neg_mean_absolute_error')
rfr_grid.fit(train_x, train_y)

print('best parameters are:-', rfr_grid.best_params_)
print('best estimator is:- ', rfr_grid.best_estimator_)
print('best mae is:- ', -1 * rfr_grid.best_score_)

MAE = 2.697

Then I fit the "best parameters" obtained to get an optimized MAE, but the results are always worse than the base model MAE.

# OPTIMIZED RFR MODEL
opt_rfr = RandomForestRegressor(random_state = 69, criterion = 'mae', max_depth = None,
                            max_features = 'auto', min_samples_leaf = 10, n_estimators = 100)
opt_rfr_pipe = Pipeline(steps = [('rfr_preproc', preproc), ('opt_rfr_model', opt_rfr)])
opt_rfr_pipe.fit(train_x, train_y)
opt_rfr_pred = opt_rfr_pipe.predict(test_x)
opt_rfr_mae = mean_absolute_error(test_y, opt_rfr_pred)

MAE = 2.496

Not just once but every time and in most of the models (linear regression, random forest regressor)! I guess there is something fundamentally wrong with my code or else this problem wouldn't arise every time. Any idea what might be causing this?

Improve this question
$\endgroup$
19
  • $\begingroup$ In this case, maybe the default parameters are the best. And your forgot to include rfr_model__min_samples_leaf default parameter which is 1. Try again including it and you may have the same and consistent answer your are looking for. You do not need to fit again the best parameters, they are already fitted. See best_estimator_ property. ANd also you do not show code for the base estimator, please show it so we can check also. $\endgroup$
    – Malo
    Jul 5, 2021 at 22:09
  • $\begingroup$ Including the default parameter values works for Random Forest regressor but not for Linear Regression and Decision Tree regressor. I still get worse performance in both the models. Also one clarification, what do you mean by "you do not need to fit again best parameters, they are already fitted". Also i've included the best_estimator_. $\endgroup$
    – spectre
    Jul 6, 2021 at 5:43
  • $\begingroup$ In last 4 lines: opt_rfr_pipe = ... and opt_rfr_pipe.fit... could be replaced by opt_rfr_pipe = rfr_grid.best_estimator_ which is fitted and ready to use for prediction. $\endgroup$
    – Malo
    Jul 6, 2021 at 6:11
  • $\begingroup$ One possible cause is overfitting: when tuning between many combinations of parameters, it's possible that one combination happens to fit the data better by chance. This is more likely in case the dataset is small. $\endgroup$
    – Erwan
    Jul 6, 2021 at 12:40
  • 1
    $\begingroup$ Does this answer your question? Is it possible to get worse model after optimization? $\endgroup$
    – Ben Reiniger ♦
    Jul 6, 2021 at 14:11

1 Answer 1

Reset to default
0
$\begingroup$

Apparently the reason I was getting worse performance was because I was using cross validation during HP tuning but not when I built the base model. Hence the issue.

Another mistake was not scaling my data!

Typical noob mistakes!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.