1
$\begingroup$

I have a binary classification task with substantial class imbalance (99% negative - 1% positive). I want to developed a Random Forest model to make prediction, and after establishing a baseline (with default parameters), I proceed to hyperparameter tuning with scikit-learn's GridSearchCV.

After setting some parameters (e.g. max_depth, min_samples_split, etc.), I noticed that the best parameters, once GridSearch was done, are highest max parameters (max_depth) and the smallest min parameters (min_samples_split, min_samples_leaf). In other words, GridSearchCV favored the combination of parameters that fits most closely to the training set, i.e. overfitting it. I always thought that cross-validation would protect from this scenario.

Therefore, my question is 'What is the point of GridSearch if the outcome is overfitting?' Have I misunderstood its purpose?

My code:

rf = RandomForestClassifier(random_state=random_state)

param_grid = {
    'n_estimators': [100, 200],
    'criterion': ['entropy', 'gini'],
    'max_depth': [5, 10, 20],
    'min_samples_split': [5, 10],
    'min_samples_leaf': [5, 10],
    'max_features': ['sqrt'],
    'bootstrap': [True],
    'class_weight': ['balanced']
}

rf_grid = GridSearchCV(estimator=rf,
                       param_grid=param_grid,
                       scoring=scoring_metric,
                       cv=5,
                       verbose=False,
                       n_jobs=-1)

best_rf_grid = rf_grid.fit(X_train, y_train)
```
$\endgroup$

2 Answers 2

0
$\begingroup$

Ideally, I would address the extreme imbalance first to get a more reliable score, try oversampling or under (I prefer over).

Then try with conservative parameters, like max_depth only until 10.

I prefer to use XGBoost over Random Forest though, it has an early stopping parameter to help prevent overfitting, although one still has to be conservative especially in the max_depth.

Then I use Optuna for hyperparameter tuning, it can also be used for Random Forest and is so much faster than GridSearchCV (especially with a huge dataset).

$\endgroup$
1
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Dec 6, 2022 at 11:20
0
$\begingroup$

You have a case of extreme imbalanced dataset. So until and unless you address that problem, whatever you do will not be much effective.

Here are some things you can try to address data imbalance:

1.) Oversampling or Undersampling. imblearn can help you with that.

2.) Ensemble learning models : Using bagging or boosting models for imbalanced dataset. You are already using bagging model (random forest). You could also try the balancedbaggingclassifier present in imblearn library. Here is the documentation for that. Also try some boosting models like XgBoost, CatBoost etc.

3.) Choose the right metric for evaluation: You have not mentioned which evaluation metric you are using. I would suggest f1_score but you can also use precision_score and recall_score depending on your problem statement.

4.) Get more data. This might be the most effective of all. Get more data specially for the minority class which will decrease the imbalance effect.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.