I am trying to find a best fit line f(x) = ?
for a random set of x,y coordinates.
Linear Regression with polynomial features works well for around 10 different polynomials but beyond 10 the r squared actually starts to drop!
If the new features are not useful to the Linear Regression I would assume that they would be given a coefficient of 0 and therefore adding features should not hurt the overall r squared.
I reproduced this problem when housing price predictions when creating a large amount of interaction features.
I have my python code below:
Create Random Data
import numpy as np
import matplotlib.pyplot as plt
def pol(x):
return x * np.cos(x)
x = np.linspace(0, 12, 100)
rng = np.random.RandomState(1234)
rng.shuffle(x)
x = np.sort(x[:25])
y = pol(x) + np.random.randn(25)*2
plt.scatter(x, y, color='green', s=50, marker='.')
plt.show()
Regress and Check Each R Squared
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
for p in range(1,30):
plot_range = [i/10 for i in range(0,120)]
poly = PolynomialFeatures(p)
X_fin = poly.fit_transform([[samp] for samp in x])
X_fin_plot = poly.fit_transform([[samp] for samp in plot_range])
reg = LinearRegression().fit(X_fin, y)
from sklearn.metrics import mean_squared_error, r2_score
print(p,r2_score(y, reg.predict(X_fin)))
Display Last Regression Line
plt.scatter(x, y, color='green', s=50, marker='.')
plt.plot(plot_range,reg.predict(X_fin_plot))
plt.show()
I also have two plots to compare. The first is with 10 polynomial features and the second is with 40. Notice how the second misses the majority of the first points.