This is really weird, I have a real simple test dataset and built a really simple linear model on it:

smallData = data.head(1000)
# smallData = data

y = smallData['points'].values
X = smallData.drop(['points','country', 'description'], axis=1)

X = pd.get_dummies(X, columns=['designation', 'province', 'region_1', 'region_2', 'variety', 'winery', 'taster_name', 'taster_twitter_handle', 'title'])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

regressor = LinearRegression()  
regressor.fit(X_train, y_train)

y_pred = regressor.predict(X_test)

It actually works really fine, as long as I only train it with ~200 rows of data. Once I have 300 rows and more, it returns COMPLETELY wrong predictions for some rows. Like, my scale is 80 to 100, and the prediction says it should be five million. That seems slightly off, especially there is no value of five million in the training data, 100 is the highest.

My data has quite a lot of columns due to the dummy encoding. It seems like it's overfitting?

What can I do to only get reasonable predictions between 80 and 100?


2 Answers 2


This is called (perfect) collinearity and happens e.g. when the reference category is included in the set of dummy variables. Use pd.get_dummies(..., drop_first=True) to avoid this.


The training of a linear regression model is very basic, it simply assigns coefficients to every feature according to what is seen in the training set. As a consequence it is very sensitive to outliers and overfitting indeed: for instance if a feature appears in the test set with a value unseen in the training set, it can throw the predicted value far from the target range. The fact that the training set doesn't contain any such value doesn't matter, since it's caused by the coefficients of the features.

In your experiment the difference is likely due to chance, not to the size of the training set (i.e. the distribution is simply different between the experiments). It's likely that if you repeat the 200 rows experiment with different random splits some of them are going to produce wrong results as well.

The more features there are, the less likely the training set is going to cover the whole range of possibilities. So reducing the number of feature would minimize the risk of overfitting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.