1
$\begingroup$

Let's scope this to just classification.

It's clear that if you fully grow out a decision tree with no regularization (e.g. max depth, pruning), it will overfit the training data and get full accuracy down to Bayes error*.

Is this universally true for all non-parametric methods?

*Assuming the model has access to the "right" features.

$\endgroup$
1
  • $\begingroup$ there is no known mathematical result that applies to "non-parametric models" in general. However it seems to be the case $\endgroup$ – Nikos M. Jul 16 '20 at 12:10
1
$\begingroup$

No - non-parametric methods only means that the method does not assume a function form of the data. There are non-parametric methods such as Random Forest that do not always overfit. In fact nonparametric methods could underfit, it could lack the ability to fit the training data. An example of this would be a decision stump.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.