Python: How to test a RandomForest regression model for Overfitting?

Question

I'm a beginner in this area so maybe I'm doing something wrong here. I'm using RandomForest for a regression model and wanted to see if my model is overfitting. Here is what I did:

EDIT:

I use GridSearchCV for hyperparameter tuning:

rf = RandomForestRegressor(random_state=0)
rf_params = {'n_estimators': [100, 500, 1000], 'max_depth': [3, 6, 9, None],
             'min_samples_leaf': [2, 5, 10], 'max_features':['auto', 'sqrt']}

gs_rf = GridSearchCV(rf, rf_params, scoring = 'neg_mean_absolute_error', cv = 10)

gs_rf.fit(x_train, y_train.values.ravel())
b = gs_rf.best_params_

Then I create a RandomForestRegressor with those parameters:

RF = RandomForestRegressor(n_estimators=b['n_estimators'], max_depth=b['max_depth'], min_samples_leaf=b['min_samples_leaf'], max_features=b['max_features'], random_state=0)

Then I fit the model using the train dataset:

model = RF.fit(x_train, y_train.values.ravel())

Then I predict with the test dataset:

y_pred = model.predict(x_test)

Then I did the exact same with x_train instead of x_test:

y_pred = model.predict(x_train)

Here are the results that I achieve:

Test Data:
MAE: 15.11
MAPE: 26.98%

Train Data:
MAE: 6.17
MAPE: 10.97%

As you can see there is a pretty significant difference. Do I have a big problem with overfitting or am I doing something wrong when using x_train to predict?

Any help is much appreciated!

Answer 1

Predicting on x_train and comparing the error with the prediction error of x_test is a good approach. Here, it seems that you are overfitting since you model is better to predict what it learned than new data. It would be a good thing to use a cross-validation approach to ensure that the problem does not come from your split.

Here are some leads I would follow:

• Apply a cross-validation to evaluate your model
• If using cross-validation in hyperparameters tuning, it is important to keep a validation dataset to evaluate your final model after the tuning. Doing that, you ensure to test your model on unseen data.
• Have a look at your hyperparameters. You could for instance decrease max_depth, that may lead to reduce overfitting.
• Plot your outputs: plot the excepted values and the predicted ones. MAE is the Mean Absolute Error, so you may discover unexpected effects (such as your model having high troubles to predict specific range values and not others)

Hope it helps.

Note
I would not say overfitting is a problem, rather its consequences could be a problem. Overfitting occurs when the model learns too much from the training data and so does not generalize well. If you constructed your pipeline well, the metric that should lead your performance is the score you choose (in your case MAE)