My lecture slide told me that if we don't prune the regression tree, then the tree likely to over-fit. So, I wonder why would that happen? Is that because if the tree grows too large, we would end up with very little instances on each leaf nodes of the tree so the estimated mean value on each leaf node will be not accurate?
2 Answers
$\begingroup$
$\endgroup$
Overfitting means that a model is giving a good fit on a dataset (whatever the measure you use to assess fit), but this is not a general case (i.e. when new data comes in or on another dataset, the error will explode. Or said otherwise, the model variance is high).
In the case of trees, adding a node to a leave based on one feature should be done only if the feature really brings information at this level. The feature could be random though and this would deteriorate greatly the fit.
As a simplistic example on a classification task; if we want to sort out apples and oranges based on some features, including one of the features that is a value, 0 or 1 chosen at random. If it happens that in our dataset the values 1 correspond in 80% of the time to apples, then we would be tempted to add a node saying "if value is 1, then apple", but you can see that this is absolutely not a generality: the tree wouldn't fit at all to another dataset. Hence this node shouldn't be added, i.e. the tree should be pruned.
$\begingroup$
$\endgroup$
If the tree is free to grow as it wishes, it can learn rules for specific training observation rather than learn generic rules for unseen data point because the objective of the decision tree is to classify well training point, not predict well unseen data. That is what means overfitting i.e. learn well in training set but predict pretty bad into new data
