However, studying polynomial regression is a bit of a challenge having some questions about the process. I understand the idea that fitting a curviliear "line" could follow the data more precisely. But seeing the following Python code in scikit-learn:
from sklearn.linear_model import LinearRegression from sklearn.preprocessing import PolynomialFeatures x = 2 - 3 * np.random.normal(0, 1, 20) y = x - 2 * (x ** 2) + 0.5 * (x ** 3) + np.random.normal(-3, 3, 20) x = x[:, np.newaxis] y = y[:, np.newaxis] polynomial_features= PolynomialFeatures(degree=2) x_poly = polynomial_features.fit_transform(x) model = LinearRegression() model.fit(x_poly, y) y_poly_pred = model.predict(x_poly)
suggests for me that here we "just" projecting the original features into 2nd degree new polynomial features, while still fitting the simple (straight) line on it. Still, we get the following solution with magically curved line:
Here arise my questions:
- How could the straight line in figure #1 become a curved one in figure #2 when we just introduce some new features but still fit the original LinearRegression model? A cannot see why couldn't the same estimator found curve in case 1 when it is able to find it in case 2? There is no hint for the estimator in the syntax in case 2 that "ok, let's apply a curve now instead of a line", right?
- I read that polynomial regression is still linear, what is the exact interpretation of this statement? "Linear" means no straight line but any curve, is this what this want to say? If so, what is "nonlinear"?
- Additionally, having read about penalty terms for polynomial regressions, I read that the introduction of higher order features has the effect that coefficients tend to grow along the magnitude. This is illustrated here:
I cannot figure out here why does coefficients for a specific same feature (like x_1) increase just because there are more additional polynomials (e. g. x_2, x_3 etc) while stay low just being alone? Quite confusing.