I am trying to find a best fit line
f(x) = ? for a random set of x,y coordinates.
Linear Regression with polynomial features works well for around 10 different polynomials but beyond 10 the r squared actually starts to drop!
If the new features are not useful to the Linear Regression I would assume that they would be given a coefficient of 0 and therefore adding features should not hurt the overall r squared.
I reproduced this problem when housing price predictions when creating a large amount of interaction features.
I have my python code below:
Create Random Data
import numpy as np import matplotlib.pyplot as plt def pol(x): return x * np.cos(x) x = np.linspace(0, 12, 100) rng = np.random.RandomState(1234) rng.shuffle(x) x = np.sort(x[:25]) y = pol(x) + np.random.randn(25)*2 plt.scatter(x, y, color='green', s=50, marker='.') plt.show()
Regress and Check Each R Squared
from sklearn.linear_model import LinearRegression from sklearn.preprocessing import PolynomialFeatures for p in range(1,30): plot_range = [i/10 for i in range(0,120)] poly = PolynomialFeatures(p) X_fin = poly.fit_transform([[samp] for samp in x]) X_fin_plot = poly.fit_transform([[samp] for samp in plot_range]) reg = LinearRegression().fit(X_fin, y) from sklearn.metrics import mean_squared_error, r2_score print(p,r2_score(y, reg.predict(X_fin)))
Display Last Regression Line
plt.scatter(x, y, color='green', s=50, marker='.') plt.plot(plot_range,reg.predict(X_fin_plot)) plt.show()