1
$\begingroup$

I'm tuning a random forest in python and am wondering if/why my model is overfit. The dataset is described below:

  • 1700 Positive Cases / 54000 total cases ~ 3.2% (unbalanced)
  • 50 Numerical Features,~450 label/hot encoded features (post data reduction)
  • 10Fold CV using 85% of data, with 15% hold out for final test
  • Classification Metrics = AUC or F1 (as data is imbalanced)

The results I get tend to suggest using very deep trees i.e depth 18 with no restriction on number of samples per split = 2(default). In this case, Train AUC was 99.9% , Max Test AUC was 84%. My scores are also almost monotonically increasing in max depth of trees. Given the results and how deep the trees are - I suspect the model is overfit? If this is the case then why would I not observe some sort of out of sample reduction in AUC as depth and min_samples_split typically constrain the random forest? Or have I overlooked anything in tuning?

My ranges in CV Grid Search are more or less:

  • n_estimates : range(100,1000,by=100)
  • max_features : {sqrt(p),0.3,0.4,0.5}
  • max_depth : range(2,20,by=1)
  • min_samples_split : range(2,50,by=1)
  • class_weights : {balanced,None}

Thanks

$\endgroup$
0
$\begingroup$

A drop in performance between train and test datasets is a sign of overfitting.

Given the extremely unbalanced data, passing sample_weight argument to RandomForest().fit() to rebalance the classes should help.

$\endgroup$
  • $\begingroup$ I'm using class_weights = "balanced" to take into account imbalances in the dataset and penalize the cost function. It seems the two are related per stackoverflow.com/questions/32492550/… $\endgroup$ – Nahyyz Jun 20 '18 at 3:15
-1
$\begingroup$

Random Forests don't overfit, the more depth you add, the more accuracy and less performance you will get.

$\endgroup$
  • $\begingroup$ Using deep trees is the problem, you have to increase the depth of your forest instead. Random Forests use very simple trees but thousands or 10th of thousands of them for that they can't overfit: Read the paper instead of downvoting.. projecteuclid.org/download/pdf_1/euclid.aos/1032181157 $\endgroup$ – Eugen Jun 20 '18 at 1:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.