I'm having a hard time wrapping my head around the idea that Linear Models can use polynomial terms to fit curve with Linear Regression. As seen here.
Assuming I haven't miss-understood the above statement, are you able to achieve good performance with a linear model, trying to fit, say a parabola in a 3d space?
Actually, a model such as $Y = b_0 + b_1X + b_2X^2$ is not a 3D parabola, but a 2D parabola. There are only two variables ($Y$ and $X$), in other words, the function is still $Y = f(X)$.
A 3D parabola would be a paraboloid, thus the model would be $Y = b_0 + b_1X + b_2X^2 + b_3Z + b_4Z^2$, and a function of the type $Y = f(X,Z)$.
Alternatively, there are mixed models such as $Y = b_0 + b_1X + b_2X^2 + b_3Z$ , which would be a Parabolic Cylinder in 3D and also $Y = f(X,Z)$.
As @Evator mentioned, all these models are in fact linear models, where the term linear is referred to the coefficients, not the variables. Thus, linearity in variables is different from linearity in coefficients. These models fit quite well but sometimes there is a cost: multicollinearity and increased variance.
Let's say you want to plot the parabola $x²$ with a linear model. This won't work on it's own, as a standard linear model can only create a function of the form $ax+b$. So what you can do is, you not only pass the value of $x$ to the model, but also the value of $x²$. Now, given $x$ and $x²$, a linear model with these two inputs can be made, which fits the function $x²$. E.g. $0\cdot x+1\cdot x²+0$. You might say this is not a linear model anymore, but it is, because it is not the model that is squaring the value $x$, it is the input that has been squared before passing it to the model. So technically it is a linear model with 2 inputs. With this trick, you can keep advantages of linear models (e.g. it is convex for the mean square error loss). By transforming your input data in any dimension space, you can use linear models to regress or classify your data.
You can also take the log10() or log() of your class/predictor variable to linearize any nonlinear relationship. This would be a little bit more general than passing x² to the model as suggested in the accepted answer.
Obviously, this trick would not work if you have negative values or 0 in your outcome variable. But then you can add +1 or + <some constant> to your outcome to get rid of these types of numeric problems. The predictions would be off by some constant. You can correct for this later.
