Linear Regression with vs without polynomial features

Question

I have a conceptual question about why (processing power/storage aside) would you ever just use a regular linear regression without adding polynomial features? It seems like adding polynomial features (without overfitting) would always produce better results. I know linear regression can fit more than just a line but that is only once you decide to add polynomial features correct? My experience is with python using sklearn's libraries.

Answer 1

Occam’s razor principle [ref]:

Having two hypotheses, that has the same empirical risk (here, training error), a short explanation (here, a boundary with fewer parameters) tends to be more valid than a long explanation.

Hence, complexity must add additional comparable accuracy. Otherwise, we should go with the simple model

Let's say you have 50-50 chance of Linear and Non-Linear data. With your approach, you will end up using regularization in 50% of the model. First, make a complex model and then regularize to make it simple.
Overall, you will increase the complexity which might impact future activities e.g Maintenance

Interpretability
Many businesses demand a reason for the decision e.g. Credit card related decision. Linear regression will give you a simple explanation for each feature

Also, read these SE answers

Answer 2

Most of the input data we face everyday is linear or can be made linear after some transformation. And Linear models are comparatively very easy to learn and generalize. You can fit add polynomial features but if your data is linear then there is a great chance that your model will overfit. So unless you know certainly that your data is not linear , you should use linear models. If linear models don't work well then you can consider adding polynomial features.

Answer 3

Linear regressions without polynomial features are used very often. One reason is that you can see the marginal effect of some feature directly from the estimated coefficient(s).

Say you have a model $y_i = \beta_0 + \beta_1 x_i + u_i$, $\beta_0$ and $\beta_1$ describe the intercept and slope of a linear function. This is often used to get a "robust" idea of how $x$ is related to $y$ (usually in a multivariate setting, so "many" $x$). It is of corse necessary to check if a linear fit is an "okay" approximation of the data generating process.

Adding polynomial features ($x^2$, $x^3$,...,$x^n$) often helps to achieve a better fit but also increases complexity. When you want to work on highly non-linear data - for which a linear approximation does not work well - you should choose another model, e.g. "generalised additive models" (GAM). The reason simply is that these models are more flexible compared to linear regression. Linear regression is "parametric", meaning that you need to propose a (ex ante unknown) functional form of the estimation equation. This is not required with other techniques, such as GAM (they are "non-parametric" if you want so say so).

The "Law of Parsimony" (aka Occam’s razor as noted by @Roshan Jha) simply says that you should choose the least complex solution for a given requirement. So in case you look for a "easy to interpret and understand" solution and/or a linear approximation works well on your data, linear regression is a good solution. Otherwise, look for a solution which is "fit" to deal with non-linearity or whatever your requirements are.