I come from a predictive modelling background, where it's common to use differential equations to model physical or chemical or biological processes.

Commonly to avoid overfitting people use AIC and penalise models for the number of free parameters they have. This is useful - because it allows us to compare models across levels of complexity - for example, a model with twice the number of free parameters may reduce the RSS, but not to justify its extra parameters.

Is there an equivalent method for Decision trees? Or for neural networks? In either case, both sets of models can always get better, when you add complexity, and is there an ML package that allows you to build models of increasing complexity whilst penalising them for increasing complexity?

  • 1
    $\begingroup$ As far as I'm aware with decision trees the complexity is controlled by the hyper-parameters (for example maximum depth, number of instances by leaf) and/or options such as pruning. By training a model with the same data but different hyper-parameters, one can observe whether more complexity improves performance. So one can select the optimal balance of complexity/performance, which is more or less equivalent to what you describe. $\endgroup$
    – Erwan
    Aug 18, 2021 at 16:04
  • $\begingroup$ thanks, thats helpful! $\endgroup$
    – Abijah
    Aug 18, 2021 at 16:11

1 Answer 1


Decision trees have many options for reducing overfitting.

Examples include:

  • Maximum depth
  • Minimum node size
  • Pruning

Another method to reduce overfitting is adding a cost function penalty. L1 or L2 penalty can be added to neural networks' cost function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.