Problem with basic understanding of polynomial regression

Question

I have an understanding of simple linear regression. Clear that results in a fitted line like this:

However, studying polynomial regression is a bit of a challenge having some questions about the process. I understand the idea that fitting a curviliear "line" could follow the data more precisely. But seeing the following Python code in scikit-learn:

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

x = 2 - 3 * np.random.normal(0, 1, 20)
y = x - 2 * (x ** 2) + 0.5 * (x ** 3) + np.random.normal(-3, 3, 20)

x = x[:, np.newaxis]
y = y[:, np.newaxis]

polynomial_features= PolynomialFeatures(degree=2)
x_poly = polynomial_features.fit_transform(x)

model = LinearRegression()
model.fit(x_poly, y)
y_poly_pred = model.predict(x_poly)

suggests for me that here we "just" projecting the original features into 2nd degree new polynomial features, while still fitting the simple (straight) line on it. Still, we get the following solution with magically curved line:

Here arise my questions:

• How could the straight line in figure #1 become a curved one in figure #2 when we just introduce some new features but still fit the original LinearRegression model? A cannot see why couldn't the same estimator found curve in case 1 when it is able to find it in case 2? There is no hint for the estimator in the syntax in case 2 that "ok, let's apply a curve now instead of a line", right?
• I read that polynomial regression is still linear, what is the exact interpretation of this statement? "Linear" means no straight line but any curve, is this what this want to say? If so, what is "nonlinear"?
• Additionally, having read about penalty terms for polynomial regressions, I read that the introduction of higher order features has the effect that coefficients tend to grow along the magnitude. This is illustrated here:

I cannot figure out here why does coefficients for a specific same feature (like x_1) increase just because there are more additional polynomials (e. g. x_2, x_3 etc) while stay low just being alone? Quite confusing.

Answer 1

"Linear regression" (aka. "ordinary least squares", OLS) refers to the type of estimator. Linear here means that you minimise the sum of squared residuals for a given (linear additive) regression equation. You can write a simple model:

$$y = \beta_0 + \beta_1 x_1 + u. $$

This would fit a linear function with intercept $\beta_0$ and slope $\beta_1$. So the "fitted" function would be:

$$ f(x) = \beta_0 + \beta_1 x. $$

You can "propose" basically any functional form to be fitted by the linear model, e.g. with a squared term (just add $x^2$ as feature):

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x^2_1 + u. $$

This will give you the (estimated) quadratic function:

$$ f(x) = \beta_0 + \beta_1 x + \beta_2 x^2. $$

In essence, you can add any kind of linear transformation of $x$, e.g. $log(x)$, $exp(x)$ etc.

"Non-linear" models (there is no really clear definition), would be e.g. logistic regresion, non-parametric regression, tree-based models, neural nets etc. While each coefficient $\beta$ directly gives you a "marginal effect" in linear (OLS) regression, this is not the case for the other (non-linear) models.

When you use regularisation in linear models, you simply add a penalty term to the loss function, so you minimise the sum of squared residuals given a penalty term. However, regarding your last question, I'm not quite sure what your actual question is.

Answer 2

Here I go with a worked example for answering mainly your first 2 questions, with some code based on this scikit-learn example. Let's generate a rough parabola as follows:

import numpy as np
import matplotlib.pyplot as plt

def f(x):
    """ function to approximate by polynomial interpolation"""
    return np.square(x)

# generate points used to plot
x_plot = np.linspace(-30, 30, 100)

# generate points and keep a subset of them
x = np.linspace(-30, 30, 100)
rng = np.random.RandomState(0)
rng.shuffle(x)
x = np.sort(x[:20])
y_true = f(x)

#let's include some noise so it is not a perfect parabola:
y_true = [y + np.random.randint(-30, 30, 1) for y in y_true]
y_true = np.array(y_true).reshape(len(y_true), )

We can also plot, for demonstration reasons, the regression line we expect to obtain after fitting our model on this parabola:

Now the question is indeed, how can we fit a linear model on this data? Let's add a higher dimensional feature (we expect degree 2 being enough):

# New input values with additional feature
import numpy as np
from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures(2)
poly_transf_X = poly.fit_transform(X)

If you plot it with the amazing plotly library, you can see the new 3D dataset (with the degree-2 new feature added) as follows (sorry I named 'z' the actual y values in this animated plot):

As you can see, the f(x) values (the z component in the plot) has a parabola form regarding the x values, but a linear shape regarding the x^2 values (y-axis on this animated plot)! This is the point.

This way, we expect that if we use linear regression as our algorithm for the final model on this new dataset, the coefficient of the x^2 values feature should be nearly 1, whereas the coefficient of the x values feature (the original one) should be nearly 0, as it does not explain the shape of our ground truth y values.

from sklearn.linear_model import Ridge, LinearRegression

reg = LinearRegression().fit(poly_transf_X, y_true)
reg.predict(poly_transf_X)
score = reg.score(poly_transf_X, y_true)

And we can finally plot both the ground truth y values and the predictions, on the 3D dataset, checking that indeed a linear regression works perfectly by adding the x^2 values feature (here I finally named the y-axis for the y values :) ):

Answer 3

It seems to me you did not understand what polynomial regression is.

1. Generally speaking, when you apply polynomial regression, you add a new feature for each power of x of the polynom. When you write : polynomial_features= PolynomialFeatures(degree=2) that means you have degree=2, that means that you add to your training dataset a new feature filled with x^2. That means that if in your first example you had : Y' = theta0 + x * theta1 now you will have : Y' = theta0 + x * theta1 + x^2 * theta2. This is a function of second degree represented by a curve.

2. Polynomial regression is linear because you have in fact Y' = Theta * X, where Theta and X are vectors. In a non linear algorithm you will have for example sigmoid(Theta * X) (used in neural networks for example).

3. One cannot say the coeficient will increase in any case, it depends on the data you have and on your model. If you want to avoid having huge differences between the values in Theta, you should apply normalization to your training and test dataset.