Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
In SKLearn's documentation on Decision Trees, they say we should pay special attention not to overfit the tree. How can we do this? I am aware that using random forests may prevent it, but how do I generally tell if it's overfitting? Can you tell by an accuracy score?
Is, for example, a 0.99 accuracy score an indicator of overfitting? Would in this case 0.95 mean not overfitting? What are some best practices in addition to a balanced input to avoid overfitting (especially with using SKLearn)?
Overfitting meaning your model is learning the noise from the data and its ability to generalize the results is very low. In this case you have a small training error but very large validation error. If you inspect (e.g. by plotting) the evolution of training and validation errors, you see that training error is always going down but validation error is goes up at some point. That is the point you need to stop training to avoid overfitting. I strongly recommend you to read this.
So, the 0.98 and 0.95 accuracy that you mentioned could be overfitting and could not! The point is that you also need to check the validation accuracy beside them. If validation accuracy is falling down then you are on overfitting zone!
What are some best practices in addition to a balanced input to avoid overfitting?
It is called Prunning. Beside general ML strategies to avoid overfitting, for decision trees you can follow pruning idea which is described (more theoretically) here and (more practically) here. In SciKit-Learn, you need to take care of parameters like depth of the tree or maximum number of leafs.
Can you tell by an accuracy score?
A general notion of gauging an overfit or an underfit is via validation curves.
How can we do this?
Not just a decision tree, (almost) every ML algorithm is prone to overfitting. One needs to pay special attention to the parameters of the algorithms in sklearn(or any ML library) to understand how each of them could contribute to overfitting, like in case of decision trees it can be the depth, the number of leaves, etc.
I am aware that using random forests may prevent it
Just a pointer that the concept of ensembling helps with better generalization of the model (which helps keep overfit in check).
