Is there a point in hyperparameter tuning for Random Forests?

Question

I have a binary classification task with substantial class imbalance (99% negative - 1% positive). I want to developed a Random Forest model to make prediction, and after establishing a baseline (with default parameters), I proceed to hyperparameter tuning with scikit-learn's GridSearchCV.

After setting some parameters (e.g. max_depth, min_samples_split, etc.), I noticed that the best parameters, once GridSearch was done, are highest max parameters (max_depth) and the smallest min parameters (min_samples_split, min_samples_leaf). In other words, GridSearchCV favored the combination of parameters that fits most closely to the training set, i.e. overfitting it. I always thought that cross-validation would protect from this scenario.

Therefore, my question is 'What is the point of GridSearch if the outcome is overfitting?' Have I misunderstood its purpose?

My code:

rf = RandomForestClassifier(random_state=random_state)

param_grid = {
    'n_estimators': [100, 200],
    'criterion': ['entropy', 'gini'],
    'max_depth': [5, 10, 20],
    'min_samples_split': [5, 10],
    'min_samples_leaf': [5, 10],
    'max_features': ['sqrt'],
    'bootstrap': [True],
    'class_weight': ['balanced']
}

rf_grid = GridSearchCV(estimator=rf,
                       param_grid=param_grid,
                       scoring=scoring_metric,
                       cv=5,
                       verbose=False,
                       n_jobs=-1)

best_rf_grid = rf_grid.fit(X_train, y_train)
```

Answer 1

Ideally, I would address the extreme imbalance first to get a more reliable score, try oversampling or under (I prefer over).

Then try with conservative parameters, like max_depth only until 10.

I prefer to use XGBoost over Random Forest though, it has an early stopping parameter to help prevent overfitting, although one still has to be conservative especially in the max_depth.

Then I use Optuna for hyperparameter tuning, it can also be used for Random Forest and is so much faster than GridSearchCV (especially with a huge dataset).

Answer 2

You have a case of extreme imbalanced dataset. So until and unless you address that problem, whatever you do will not be much effective.

Here are some things you can try to address data imbalance:

1.) Oversampling or Undersampling. imblearn can help you with that.

2.) Ensemble learning models : Using bagging or boosting models for imbalanced dataset. You are already using bagging model (random forest). You could also try the balancedbaggingclassifier present in imblearn library. Here is the documentation for that. Also try some boosting models like XgBoost, CatBoost etc.

3.) Choose the right metric for evaluation: You have not mentioned which evaluation metric you are using. I would suggest f1_score but you can also use precision_score and recall_score depending on your problem statement.

4.) Get more data. This might be the most effective of all. Get more data specially for the minority class which will decrease the imbalance effect.