Random Forest overfitting to unbalanced data set

Question

I am working on an unbalanced classification problem. I have have 2000 points which are positive, and 6000 points as -ve (chosen randomly from 100k universe of -ve points universe). Although I have ~40 features, I am using top 15 (in order of RF feature importance).

After I split my data into X_train, X_test, y_train, y_test I. I train both logistic regression (without regularisation) and RF (with hyperparameter tuning using RandomizedSearchCV with random_grid={ 'max_depth': [4,10,12],'max_features': ['auto', 'sqrt'],'min_samples_leaf': [1, 2, 4],'min_samples_split': [2, 5, 10],'n_estimators': [200, 400, 600, 800]} and RandomForestClassifier(class_weight='balanced')

My first observation is:

1. The AUC on RF is 0.99 which clearly indicates overfitting.
2. The LR also gives a fairly high AUC of 0.92 but not as high as RF
3. When I test on a completely new data set (although I do not have true labels), with some domain knowledge, I can say that the LR model gives much better results.
4. This could not be a data leakage problem because LR gives much better results.
5. Another observation is that LR probabilities are distributed to the tail end also however RF there are none after 0.8

My questions are:

1. Why RF is getting overfit in spite of hyperparameter tuning?
2. What are the options?

Answer 1

There are some options not to overfit your Random Forest Classifier given your observations:

1. Complexity of the Model (Tree Depth)

Check the max_depth parameter in the tuned RF model. A very high value might contribute to overfitting. Consider reducing it to limit the depth of the trees.

2. Number of Trees (n_estimators)

Large values of n_estimators can lead to overfitting. Experiment by reducing the number of trees to find a balance that prevents overfitting while maintaining good performance.

3. Feature Importance and Selection:

Reevaluate the features selected based on RF feature importance. Using too many features may introduce noise. Experiment with a smaller subset to assess its impact on overfitting.

4. Cross-Validation Strategy

Ensure an appropriate cross-validation strategy during hyperparameter tuning. Consider using a stratified k-fold to maintain the class distribution in each fold.

5. Class Imbalance Handling

Despite using class_weight='balanced', class imbalance might still pose a challenge. Try to address the imbalance by using oversampling, undersampling, or advanced techniques like SMOTE.

6. Ensemble Model Calibration

Random Forests can produce overly confident probability estimates. Consider calibration techniques like Platt scaling or isotonic regression. Scikit-learn's CalibratedClassifierCV can be useful for this purpose.