1
$\begingroup$

I'm building a simple linear regression model that predicts Home Price using Square Footage, Number of Bed(s), and Number of Bathroom(s).

After creating the model, I noticed that the coefficients for Square Footage and Number of Bed(s) were positive, which makes sense since Home Prices increases as Square Footage/Number of Bed(s) increases. However, the coefficient for Number of Bathroom(s) was negative, which makes no sense since Home Price does not decrease as Number of Bathroom(s) increases!

My question is this: is the negative coefficient something I can "fix" by incorporating more data into my model, or is this just a quirk in linear regression?

$\endgroup$
2
$\begingroup$

What could be happening here is multicollinearity - where some of your predictors are correlated. In this case, it is not unreasonable to think that square footage, number of beds and number of bathrooms are all positively correlated with each other. This is not a big deal if your main goal is to just predict house prices, but what multicollinearity does in essence is that it makes it 'harder' to estimate the actual coefficient values, so your interpretations of the coefficients can't carry as much weight. I suspect that using more data will indeed reduce the chance of this happening.

$\endgroup$
0
$\begingroup$

You can do something about this, the question is whether you want to. As rzch mentioned, if you just care about prediction accuracy, you might want to allow your coefficients to take "unusual values". This is certainly how you will get the lowest training loss (provided you are restricting yourself to a multi-linear model), but it might not give you your lowest test loss (you'll have to check this under cross validation).

If you want to learn something more "physical" about your data, you might consider the following, stage-wise approach.

1: For each of your M features, fit a linear model (i.e. use that one feature to best predict the target). See which of your features gives you the lowest test loss and select this one as your "best" feature.

2: Using the model fit on your best feature (according to stage 1), calculate your training residuals ($\hat{y}_{i}-y_{i}$), and for each of the (M-1) remaining features, fit a linear model to the residuals. Again, select the feature which gives you the best test loss.

3: Continue the process until there is no model you can fit which gives you an increase in test loss (or the increases in test loss become inconsequential).

This allows you to come up with the following interpretation:

Your most important feature, the feature used in the first model, is easy to interpret, the size and magnitude of the coefficient tell you how it's related to the target variable.

The size and sign of the coefficient related to your second variable tell you how that feature is related to the target, after taking into account the first variable.

At least for the first few features, this is way easier to interpret, and admittedly becomes a bit less tractable when you have a lot of variables.

This procedure will likely also be more robust to over-fitting than an out-of-the-box linear regression. If combatting over-fitting is your only concern however, methods like ridge/lasso regression are considerably less computationally intensive.

$\endgroup$
0
$\begingroup$

I think the problem may caused by correlation between your predictors and by non-linear dependiences. You could try computing Pearson $r$ linear-correlation indicator for checking each predictor-house pricing pair.

If $r \approx 1$ it means that in fact there is strong possibility that house_pricing(predictor) is linear. And growth of predictor causes growth of house pricing. If $r \approx -1$ is means that pretty same, but growth of predictor would cause drop of house pricing. If $|r|$ is low - below somewhat 0.3 0.4 then your house_pricing(predictor) is not really a linear function.

I would also check if input variables are correlated with each other. I think that number of beds and bathrooms are correlated to Square Footage. Maybe PCA (Principal Component Analysis) would help you get better (or more intuitive) results. Or maybe your data is just exceptional and getting more information would make this clean.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.