7
$\begingroup$

In SKLearn's documentation on Decision Trees, they say we should pay special attention not to overfit the tree. How can we do this? I am aware that using random forests may prevent it, but how do I generally tell if it's overfitting? Can you tell by an accuracy score?

Is, for example, a 0.99 accuracy score an indicator of overfitting? Would in this case 0.95 mean not overfitting? What are some best practices in addition to a balanced input to avoid overfitting (especially with using SKLearn)?

$\endgroup$

2 Answers 2

5
$\begingroup$

Overfitting meaning your model is learning the noise from the data and its ability to generalize the results is very low. In this case you have a small training error but very large validation error. If you inspect (e.g. by plotting) the evolution of training and validation errors, you see that training error is always going down but validation error is goes up at some point. That is the point you need to stop training to avoid overfitting. I strongly recommend you to read this.

So, the 0.98 and 0.95 accuracy that you mentioned could be overfitting and could not! The point is that you also need to check the validation accuracy beside them. If validation accuracy is falling down then you are on overfitting zone!

What are some best practices in addition to a balanced input to avoid overfitting?

It is called Prunning. Beside general ML strategies to avoid overfitting, for decision trees you can follow pruning idea which is described (more theoretically) here and (more practically) here. In SciKit-Learn, you need to take care of parameters like depth of the tree or maximum number of leafs.

$\endgroup$
1
  • $\begingroup$ >So, the 0.98 and 0.95 accuracy that you mentioned could be overfitting and could not! +1. Exactly what I wanted to say but didn't :D $\endgroup$
    – Dawny33
    Jan 18, 2018 at 10:24
2
$\begingroup$

Can you tell by an accuracy score?

A general notion of gauging an overfit or an underfit is via validation curves.

How can we do this?

Not just a decision tree, (almost) every ML algorithm is prone to overfitting. One needs to pay special attention to the parameters of the algorithms in sklearn(or any ML library) to understand how each of them could contribute to overfitting, like in case of decision trees it can be the depth, the number of leaves, etc.

I am aware that using random forests may prevent it

Just a pointer that the concept of ensembling helps with better generalization of the model (which helps keep overfit in check).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.