0
$\begingroup$

I am working on an unbalanced classification problem. I have have 2000 points which are positive, and 6000 points as -ve (chosen randomly from 100k universe of -ve points universe). Although I have ~40 features, I am using top 15 (in order of RF feature importance).

After I split my data into X_train, X_test, y_train, y_test I. I train both logistic regression (without regularisation) and RF (with hyperparameter tuning using RandomizedSearchCV with random_grid={ 'max_depth': [4,10,12],'max_features': ['auto', 'sqrt'],'min_samples_leaf': [1, 2, 4],'min_samples_split': [2, 5, 10],'n_estimators': [200, 400, 600, 800]} and RandomForestClassifier(class_weight='balanced')

My first observation is:

  1. The AUC on RF is 0.99 which clearly indicates overfitting.
  2. The LR also gives a fairly high AUC of 0.92 but not as high as RF
  3. When I test on a completely new data set (although I do not have true labels), with some domain knowledge, I can say that the LR model gives much better results.
  4. This could not be a data leakage problem because LR gives much better results.
  5. Another observation is that LR probabilities are distributed to the tail end also however RF there are none after 0.8

My questions are:

  1. Why RF is getting overfit in spite of hyperparameter tuning?
  2. What are the options?
$\endgroup$
6
  • 2
    $\begingroup$ Why you do you think that RF is clearly overfitting while LR is not based on the numbers 0.99 and 0.92? $\endgroup$ Nov 17, 2023 at 11:21
  • $\begingroup$ yes primarily thats the reason i think $\endgroup$
    – Gupta
    Nov 17, 2023 at 12:10
  • $\begingroup$ Could you please elaborate on point #5? Could you post histograms of the predicted probability values, for instance? $\endgroup$
    – Dave
    Nov 18, 2023 at 1:01
  • $\begingroup$ @Dave : I can not post the histograms as they are based on confidential data. However, got to understand that such distribution is such coz RF probabilities are not calibrated and thus this distribution. $\endgroup$
    – Gupta
    Nov 18, 2023 at 20:17
  • $\begingroup$ What is the distribution for the logistic regression predictions? $//$ Calibration would be assessed by a different kind of plot. sklearn has a plotting function in Python, and rms has one in R. $\endgroup$
    – Dave
    Nov 18, 2023 at 20:27

1 Answer 1

0
$\begingroup$

There are some options not to overfit your Random Forest Classifier given your observations:

1. Complexity of the Model (Tree Depth)

Check the max_depth parameter in the tuned RF model. A very high value might contribute to overfitting. Consider reducing it to limit the depth of the trees.

2. Number of Trees (n_estimators)

Large values of n_estimators can lead to overfitting. Experiment by reducing the number of trees to find a balance that prevents overfitting while maintaining good performance.

3. Feature Importance and Selection:

Reevaluate the features selected based on RF feature importance. Using too many features may introduce noise. Experiment with a smaller subset to assess its impact on overfitting.

4. Cross-Validation Strategy

Ensure an appropriate cross-validation strategy during hyperparameter tuning. Consider using a stratified k-fold to maintain the class distribution in each fold.

5. Class Imbalance Handling

Despite using class_weight='balanced', class imbalance might still pose a challenge. Try to address the imbalance by using oversampling, undersampling, or advanced techniques like SMOTE.

6. Ensemble Model Calibration

Random Forests can produce overly confident probability estimates. Consider calibration techniques like Platt scaling or isotonic regression. Scikit-learn's CalibratedClassifierCV can be useful for this purpose.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the elaborate answer. I used stratified k-fold and it seems to improve performance. Trying all other suggestions. $\endgroup$
    – Gupta
    Nov 19, 2023 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.