I stumbled into this blog which shows how a decision tree trained to overfit the predictions of a properly trained random forest model is able to generalize in pretty much the same way as the original random forest.

I'm interested in this as I'm implementing ML in embedded settings where a 1000 decisors RF is simply not feasible but a simpler tree with 10s of branches may be doable.

My first question: is this too good to be true? The only downside I can see is that the resulting decision tree will be very large due to the overfitting process but in any instance I assume it to be simpler than the full random forest.

Secondary question: is there anything in literature that discusses this process in more detail?


1 Answer 1


No, this decision tree is unable to "generalize in the same way as the original random forest".

The author also clearly states this in section Does this aproximation hold for unseen data?: '... the only problem is that this strategy applies strictly only on the seen/available data'. At least no guarantee.

The main use of this method is its explainability - using a simple, easily explainable model to mimic the behavior of a more complex model so to gain insight on how the complex model make decision. However, again this explainability does not hold for unseen data in general.

  • $\begingroup$ Would you then think that a decision tree trained from scratch, pruned to avoid overfitting would be a better fit? $\endgroup$
    – amiando
    May 25, 2023 at 11:02
  • $\begingroup$ @amiando that would just be the weaker version of a decision tree directly trained with original data. $\endgroup$
    – lpounng
    May 29, 2023 at 2:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.