I use a linear regression to predict house prices (https://www.kaggle.com/c/house-prices-advanced-regression-techniques/overview). My linear regression sometimes works great with R^2 of 0.8 and sometimes really sucks with R^2 of - 20000000000000 (Yes, really that bad). My data is scaled (with Min-Max) but my target value isn't. The problem appeared before scaling as well. Apparently my predictions are only a few times off, but then they are really off.

Here is my code:

df = pd.read_csv('data/results/house_prices_advanced_regression/house_prices_advanced_regression_.csv')
target = df['SalePrice']
df = df.drop('SalePrice', axis=1)
x_train, x_test, y_train, y_test = train_test_split(df, target, test_size=0.2)

# model
lr = LinearRegression().fit(x_train, y_train)

predictions = lr.predict(x_test)
rmse = mean_squared_error(y_test, predictions, squared=False)
mae = mean_absolute_error(y_test, predictions)
print('RMSE: {}'.format(rmse))
print('MAE: {}'.format(mae))
x = range(len(y_test))
plt.plot(x, predictions, 'r')
plt.plot(x, y_test, 'b')
print("R^2: {}".format(lr.score(x_test, y_test)))


'RMSE: 4.2834860573491624e+16'
'MAE: 3483821398471256.5'
'R^2: -3.358286170039772e+23'

Difference between prediction (red) and real value(blue):


Does anybody know what to do?

  • $\begingroup$ Have you checked for colinearity? It seems that you are (implicitly through the use of LinearRegression) trying to invert a singular matrix. $\endgroup$ Commented May 24, 2021 at 13:58
  • $\begingroup$ Yeah, killing multicorrelated features didn't help. It seams like it only mispredicts a few times, but then it really takes a deep jump into the wrong direction :/ $\endgroup$ Commented May 24, 2021 at 14:30

1 Answer 1


There are a number of approaches to try and improve the model but more importantly to understand what is happening.

First of all, you've displayed a plot with over 200 'features' and despite scaling the input the coefficients are in the order of 1E17!!! Edit: house prices (in dollars are in the range 1E5-E7, ie less than $10M), yet the coefficients are at least 7 orders of magnitude greater or smaller.

Therefore, two immediate steps to take either together or in combination is conduct:

  1. dimensionality reduction either using a formal method e.g. PCA or just dropping terms that have coefficients close to zero backed with some plots to help you decide.

  2. regularisation - at at the moment you have some extreme and opposite coefficients around +7.5E17 and -7.5E17. See models here https://scikit-learn.org/stable/modules/linear_model.html#linear-model that implement ways to suppress extreme coefficients. Edit: If your input data is between 0-1 ask yourself why for some parameters are at opposite ends, in effect cancelling each other out.

  3. Use some personal domain knowledge and evaluate the individual columns. Is there anything that has no bearing on the sale price? Some silly examples might the name of first person who bought it, which has no relevance to house price. Or a telephone number etc.

As we're now entering the world of 'hyper-parameter' tuning don't forget to split your data into 3 parts instead of 2.

  • $\begingroup$ Thanks for your answer. The training data ist scaled, but not the target value, which is why the coefficients are that big. Is it necessary to scale the target as well, if so, why? Scaling, in my understanding, is used to prevent the impact from pure higher values in comparison to lower values. 1. I want to compare different selection algorithms and I need a baseline with all variables. 2. I will try that. 3. I want to use a quantitive approach, this is why domain knowledge mustn't be used. What is the third part used for? $\endgroup$ Commented May 25, 2021 at 6:27
  • $\begingroup$ The third part is ultimately about removing superfluous variables using something other than statistical methods. $\endgroup$
    – fswings
    Commented May 25, 2021 at 21:34
  • $\begingroup$ For example, how many rows have an actual valid pool area value? $\endgroup$
    – fswings
    Commented May 25, 2021 at 21:37
  • $\begingroup$ Finally, the whole point of Data Science is to use domain knowledge wherever you can. Why don't want to use that? $\endgroup$
    – fswings
    Commented May 25, 2021 at 21:39
  • 1
    $\begingroup$ By the way. The Ridge Regression saved me. The coefficients were too high, which was probably due to one hot encoding of categories. I think the overweighing of variables or dependance of the variables confused the normal linear regression. $\endgroup$ Commented May 26, 2021 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.