Random Forest Classifier - KFold CV Tunes Very Deep Trees --> Overfitting?

Question

I'm tuning a random forest in python and am wondering if/why my model is overfit. The dataset is described below:

• 1700 Positive Cases / 54000 total cases ~ 3.2% (unbalanced)
• 50 Numerical Features,~450 label/hot encoded features (post data reduction)
• 10Fold CV using 85% of data, with 15% hold out for final test
• Classification Metrics = AUC or F1 (as data is imbalanced)

The results I get tend to suggest using very deep trees i.e depth 18 with no restriction on number of samples per split = 2(default). In this case, Train AUC was 99.9% , Max Test AUC was 84%. My scores are also almost monotonically increasing in max depth of trees. Given the results and how deep the trees are - I suspect the model is overfit? If this is the case then why would I not observe some sort of out of sample reduction in AUC as depth and min_samples_split typically constrain the random forest? Or have I overlooked anything in tuning?

My ranges in CV Grid Search are more or less:

• n_estimates : range(100,1000,by=100)
• max_features : {sqrt(p),0.3,0.4,0.5}
• max_depth : range(2,20,by=1)
• min_samples_split : range(2,50,by=1)
• class_weights : {balanced,None}

Thanks

Answer 1

There are signs of overfitting the data when the model's performance drops on the test set considerably.

Good feature engineering can also improves your model's performance. Considering that you have an imbalanced data set, you can look at resampling techniques for your data set such as random over/under-sampling and SMOTE. You can also extract feature importances from ensemble learners or regularized regression models like LASSO to see which features are contributing the most weight in your model's predictions. You could also apply various statistical tests such as ANOVA or chi-square square tests to gain further insights of your data. There is lots to do here.

You can also reconsider using the ROC AUC metric and choose something better tuned to a class imbalance in your data set. The ROC curve plots the True Positive Rates (TPR) on the x-axis and the False Positive Rates(FPR) on the y-axis whose formulas respectively correspond to $$TPR=\frac{TP}{TP+FN}$$ and $$FPR=\frac{FP}{FP+TP}$$ Since TPR is based only on the TP instances, the ROC curve won't measure the effects of negative instances. The AUC does does not consider the class imbalance when evaluating overall model performance, so we cannot trust it to weigh on the underrepresented class.

One way to resolve this is by using the Matthews Correlation Coefficient since it takes into account true and false positives and negatives. This helps represent the minority class a lot more during your model evaluation.

You could also look into random grid searched for your hyperparameter tuning; this method uses a fixed number of parameter settings sampled from your specified distribution. This allows you to get a general neighborhood of your parameter space where you can use a brute force grid search in a shorter parameter space. Then you can use cross validation to evaluate your model as you normally would.

This is just a handful of things to consider, there are many other things that you may find are suitable for your problem.

Answer 2

A drop in performance between train and test datasets is a sign of overfitting.

Given the extremely unbalanced data, passing sample_weight argument to RandomForest().fit() to rebalance the classes should help.

Answer 3

Random Forests don't overfit, the more depth you add, the more accuracy and less performance you will get.