1
$\begingroup$

Given a dataset and a decision tree that can be as depth as it wants, if you train the tree with all the dataset and then you test it against the same dataset and you get an accuracy that is not 100%, what can you tell about your data? One would expect a perfect accuracy as you let the tree overfit as much as wanted, so what's that difference on accuracy?

$\endgroup$
1
  • 1
    $\begingroup$ Have you made sure that the parameters of your decision tree allow the model to fully overfit? Tree depth is one you already looked at, but there may be other parameters that prevent the tree from achieving 100% training accuracy, such as the minimum number of samples in the leaf nodes. $\endgroup$
    – Oxbowerce
    Commented Mar 22, 2022 at 15:30

1 Answer 1

1
$\begingroup$

There are many possible reasons why the model might not reach perfect performance:

  • As mentioned by Oxbowerce in a comment, there are several hyper-parameters other than the tree depth which might force the model to generalize: max number of instances per leaf, pruning, etc. It's even possible that some implementations apply some generalization methods without providing parameters to control them.
  • The dataset might include contradicting evidence, i.e. several instances with the same feature values but different labels. This is more common with categorical data but it can also happen with numerical data.

Purposefully overfitting a model is an interesting experiment, but in general ML models are meant to generalize from the data, so they are not intended for just storing all the data from the training set (except lazy learners like k-NN). So there's no guarantee that they would be able to fully represent the whole training data accurately.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.