0
$\begingroup$

Though tree-based ML algorithms give us 100% accuracy on train dataset many times, but why is this not happening every time. I know this results in overfitting but why not 100% accuracy every time on the dataset using which our model is trained?

$\endgroup$
1
  • $\begingroup$ Some data cannot be accurately classified in any way. $\endgroup$ Feb 15, 2023 at 12:14

1 Answer 1

2
$\begingroup$

Multiple reasons:

  • Contradictory/inconsistent data (e.g. the training data contains the same input with different outputs).
  • Limits in the model's memorization capacity (e.g. limited tree depth).

If the model has infinite memorization capacity and the data is consistent, it should be possible to memorize the whole training data, as you hinted.

$\endgroup$
2
  • 1
    $\begingroup$ +1 People often forget that first point, but there are some data that just can’t be predicted perfectly. $\endgroup$
    – Dave
    Feb 15, 2023 at 14:04
  • $\begingroup$ @Dave I think the question refers to the model memorizing the training data, not predicting anything, just pure overfitting by memorization. It's like the experiments with neural networks memorizing random labels assigned to images. A sufficiently powerful model can memorize the training data if the data is self-consistent. In your linked answer, the data was contradictory, exactly my point: the same input appeared twice in the training data with 2 different outputs (x=3, y=3 and x=3, y=4). $\endgroup$
    – noe
    Feb 15, 2023 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.