I've built a Random Forest model that classifies into four categories based on around 10 input features.
To test the accuracy, I performed 5-fold stratified cross validation using the
sklearn.model_selection.StratifiedKFold method for a number of different maximum depths.
I then examined the test and train accuracy for each of these, amongst some other metrics like one-vs-one AUC. I was expecting as the maximum depth increased, the test and training accuracy would both increase, up to a point where the model starts to overfit, from which point the test accuracy would start to decrease while train accuracy continues to increase.
What I've seen instead is the test accuracy just plateauing around a certain value, even when maximum depth is increased to 100s.
I'm confused about why it's not behaving as I thought, and how to pick the optimal depth. Do I have a misunderstanding about overfitting in Random Forests?
Max_depth Test_acc Train_acc 3 0.54 0.58 5 0.65 0.67 7 0.82 0.82 9 0.84 0.85 11 0.87 0.91 15 0.87 0.95 20 0.87 0.96 50 0.87 1.0 100 0.87 1.0