1
$\begingroup$

When one tries to look up concepts such as overfitting and underfitting, the most common thing that pops up is polynomial regression. Why is polynomial regression often used to demonstrate these concepts? Is it just because it can be easily visualised like the graphs here:

https://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html

But then most ml algorithm such as kmeans clustering can also be used. Then why is it usually polynomial regression only? Are there any other similar algorithms that could be used?

$\endgroup$
2
  • $\begingroup$ almost all algorithms can be used as all have these drawbacks, yet polynomial regression is easily explainable rather than a neural network or decision tree $\endgroup$ – Nikos M. May 7 at 20:46
  • $\begingroup$ See also your related stats.se question: stats.stackexchange.com/q/523207/232706 $\endgroup$ – Ben Reiniger May 8 at 18:11
1
$\begingroup$

Linear Algebra tells us that N linearly independent vectors spans all of N dimension space. In a regression setting this translates into the fact that if you have N observations and N features per observation then your regression model has a fair chance of achieving a 100 % accuracy on the NxN training data. And your chances are even better if the NxN feature set is mostly composed of noise because N randomly generated N-dimension vectors are linearly independent.

The high accuracy on the train set is because the model has overfit on the random noise in the data. Such models almost never generalize well on an out of sample test set.

So, what's happening in polynomial regression is that as you add newer features, they may add more noise (potentially useless info) but very capable of being used by the model to explain the variance in the train set but never on the test set.

That's why it is the ideally choice because it allows you to easily add newer features from existing ones and demonstrate overfitting on a train set.

$\endgroup$
1
$\begingroup$

Another example you could use is the separation boundaries in decision trees classification problem. In the picture below, you can see that the training error continues to go down (lower the better) when max_depth increases, whereas the testing error is not as good. This is because the model has carved out specific pink areas (for 'x') as the separation boundary based on the training data. When these boundaries are applied to another set of data, they become poor separation boundaries. So these boundaries cannot be generalised to other testing data sets.

separation boundaries in decision trees

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.