What would be a good way to go around finding the best depth for a DecisionTree (in SKLearn)? How can I tell if I've gone too deep and am overfitting? I know I can find the best parameters with f.e. GridSearchCV, but the best score might not mean I get the best classifier as I may overfit the tree to the data.
No! the best score on validation set means you are not in overfitting zone. As explained in my previous answer to your question, overfitting is about high score on training data but low score on validation. So to avoid overfitting you need to check your score on Validation Set and then you are fine. There is no theoretical calculation of the best depth of a decision tree to the best of my knowledge.
So here is what you do:
- Choose a number of tree depths to start a for loop (try to cover whole area so try small ones and very big ones as well)
- Inside a for loop divide your dataset to train/validation (e.g. 70%/30%)
- Each time train your decision tree with that depth on training data and test it on the validation set, then keep the validation error (you can also keep the training error)
- Plot the validation error (you can combine it with evolution of training error to have a prettier plot for understanding!)
- Find the global minimum of validation error.
- Then you can narrow your search in a new for loop according to the value you found to reach a more precise value
Actually there is the possibility of overfitting the validation set.
This because the validation set is the one where your parameters (the depth in your case) perform at best, but this does not means that your model will generalize well on unseen data.
That's the reason why usually you split your data into three set: train, validation and test.
You train on the training set, tune the parameters on the validation set, and finally, when you are happy with the parameters, you test your model as a whole with the test set. And usually, the error on the test set will be higher than the error on the validation test.
If this difference is small, you accept the model. But if it's big, you need to act:
1) reorganizing the three sets, because maybe you have a variance problem between the sets;
2) adding some penalty on your model, acting on the regularization parameters; or lowering the depth of the trees, if you are interested only on it