This is really weird, I have a real simple test dataset and built a really simple linear model on it:
smallData = data.head(1000) # smallData = data y = smallData['points'].values X = smallData.drop(['points','country', 'description'], axis=1) X = pd.get_dummies(X, columns=['designation', 'province', 'region_1', 'region_2', 'variety', 'winery', 'taster_name', 'taster_twitter_handle', 'title']) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0) regressor = LinearRegression() regressor.fit(X_train, y_train) y_pred = regressor.predict(X_test)
It actually works really fine, as long as I only train it with ~200 rows of data. Once I have 300 rows and more, it returns COMPLETELY wrong predictions for some rows. Like, my scale is 80 to 100, and the prediction says it should be five million. That seems slightly off, especially there is no value of five million in the training data, 100 is the highest.
My data has quite a lot of columns due to the dummy encoding. It seems like it's overfitting?
What can I do to only get reasonable predictions between 80 and 100?