0
$\begingroup$

I'm a beginner in this area so maybe I'm doing something wrong here. I'm using RandomForest for a regression model and wanted to see if my model is overfitting. Here is what I did:

EDIT:

I use GridSearchCV for hyperparameter tuning:

rf = RandomForestRegressor(random_state=0)
rf_params = {'n_estimators': [100, 500, 1000], 'max_depth': [3, 6, 9, None],
             'min_samples_leaf': [2, 5, 10], 'max_features':['auto', 'sqrt']}

gs_rf = GridSearchCV(rf, rf_params, scoring = 'neg_mean_absolute_error', cv = 10)

gs_rf.fit(x_train, y_train.values.ravel())
b = gs_rf.best_params_

Then I create a RandomForestRegressor with those parameters:

RF = RandomForestRegressor(n_estimators=b['n_estimators'], max_depth=b['max_depth'], min_samples_leaf=b['min_samples_leaf'], max_features=b['max_features'], random_state=0)

Then I fit the model using the train dataset:

model = RF.fit(x_train, y_train.values.ravel())

Then I predict with the test dataset:

y_pred = model.predict(x_test)

Then I did the exact same with x_train instead of x_test:

y_pred = model.predict(x_train)

Here are the results that I achieve:

Test Data:
MAE: 15.11
MAPE: 26.98%

Train Data:
MAE: 6.17
MAPE: 10.97%

As you can see there is a pretty significant difference. Do I have a big problem with overfitting or am I doing something wrong when using x_train to predict?

Any help is much appreciated!

$\endgroup$
1
$\begingroup$

Predicting on x_train and comparing the error with the prediction error of x_test is a good approach. Here, it seems that you are overfitting since you model is better to predict what it learned than new data. It would be a good thing to use a cross-validation approach to ensure that the problem does not come from your split.

Here are some leads I would follow:

  • Apply a cross-validation to evaluate your model
  • If using cross-validation in hyperparameters tuning, it is important to keep a validation dataset to evaluate your final model after the tuning. Doing that, you ensure to test your model on unseen data.
  • Have a look at your hyperparameters. You could for instance decrease max_depth, that may lead to reduce overfitting.
  • Plot your outputs: plot the excepted values and the predicted ones. MAE is the Mean Absolute Error, so you may discover unexpected effects (such as your model having high troubles to predict specific range values and not others)

Hope it helps.

Note
I would not say overfitting is a problem, rather its consequences could be a problem. Overfitting occurs when the model learns too much from the training data and so does not generalize well. If you constructed your pipeline well, the metric that should lead your performance is the score you choose (in your case MAE)

$\endgroup$
11
  • $\begingroup$ thanks a lot for your reply! I used cross-validation in the GridSearchCV for the Hyperparameter Tuning. Is that what you mean? Or is there somewhere else I should use the cross-validation? max_depth was set to 'None' through the Hyperparameter Tuning, for which I used: 'max_depth': [1, 3, 5, 7, 9, 11, None]. Should I change something here? Once again, big thanks for helping out! $\endgroup$ – 0009 Dec 15 '20 at 11:13
  • $\begingroup$ To follow up on this: I now tried it with setting max_depth=9. Now I get the following results: Test Data: MAE: 16.01 MAPE: 28.19% Train Data: MAE: 13.11 MAPE: 20.54% So while the overfitting problem seems to have gotten decreased, I also achieve slighty worse results with the Test Data. Is this desirable? $\endgroup$ – 0009 Dec 15 '20 at 11:39
  • $\begingroup$ No it is not desirable. Overfitting is one problem, but sure the goal is to build a model that performs well-enough according to our needs. I also added a bullet point to use a third dataset to evaluate the final model after hyperparameters tuning $\endgroup$ – etiennedm Dec 15 '20 at 12:37
  • $\begingroup$ thanks, wouldn't the test dataset be an unseen dataset? or am I mistaken here? could you maybe provide a link on how using a third dataset would work? $\endgroup$ – 0009 Dec 15 '20 at 14:05
  • $\begingroup$ Sure, you could read this. Notice that I mixed validation and test dataset... However, in your case, because you did the hyperparameters tuning using all the data, you don't have this unseen test dataset, if I understand correctly how you proceed. $\endgroup$ – etiennedm Dec 15 '20 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.