1
$\begingroup$

It seems to be generally acknowledged that decision trees have low prediction accuracy. Is there a concise explanation for why they have low accuracy?

I've read this so much, I've accepted it to be true, but I realize I don't have any intuition as to why it's true.

As an example, here's an excerpt from Elements of Statistical Learning (page 352):

Trees have one aspect that prevents them from being the ideal tool for predictive learning, namely inaccuracy. They seldom provide predictive ac- curacy comparable to the best that can be achieved with the data at hand.

Or on Wikipedia, under the heading Disadvantages of Decision Trees: "They are often relatively inaccurate. Many other predictors perform better with similar data. "

$\endgroup$
  • $\begingroup$ Could you send an article/source where "low accuracy" is stated? I am just curious. $\endgroup$ – fuwiak Apr 6 at 22:13
  • $\begingroup$ @fuwiak I updated the question with a couple examples. $\endgroup$ – Nick Koprowicz Apr 7 at 1:07
0
$\begingroup$

Your question is right. First a common misconception, Decision trees are deterministic and extremely greedy. A random forest is not a decision tree, it as an ensemble of decision trees selected in a way to avoid the potential pitfall of a decision tree.

If you continue reading in both of your referencees.

In wikipedia

They are often relatively inaccurate. Many other predictors perform better with similar data. This can be remedied by replacing a single decision tree with a random forest of decision trees...

Because they are greedy and deterministic if you add one row more or take one out the result can be different, also that they tend to overfit. That is my understanding of low accuracy in this sentence.

In elements of statistical learning

Trees have one aspect that prevents them from being the ideal tool for predictive learning, namely inaccuracy. They seldom provide predictive accuracy comparable to the best that can be achieved with the data at hand. As seen in Section 10.1, boosting decision trees improves their accuracy, often dramatically. A

Because they are greedy and deterministic they don't normally give their best result. That is why random forest and gradient boosting appeared and they are extremely good. They replace this pitfall of decision trees.

Also have a look at the No Free Lunch theorem.

In short your question is right, and that problem has been solved historically with random forest and gradient boosting.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks for your response. So the real problem is that they're greedy and deterministic? Can you say more about what you mean by deterministic? $\endgroup$ – Nick Koprowicz Apr 26 at 19:12
  • 1
    $\begingroup$ Yes! Given a fixed dataset, do they always produce a tree with the same structure: ai.stackexchange.com/questions/11576/… $\endgroup$ – Carlos Mougan Apr 27 at 7:44
3
$\begingroup$

It's not true in general. Decision trees tends to overfit in comparison to other algorithms, which provide too low accuracy. But if you use a decision tree in the right way i.e you prepare data in the proper format, use feature selection and perform k-fold cross-validation everything should be ok.

I am sure that you misread it. There is no reason why DT could be a much worse algorithm compared to others.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I see that people are upvoting this answer, but I guess my question still stands - in the quotes from Elements and Wikipedia, do you think they're only referring to the tendency of Decision Trees to overfit? If so, why don't they say so? $\endgroup$ – Nick Koprowicz Apr 8 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.