4
$\begingroup$

I have a dataset that has the following columns:

enter image description here

The variable I'm trying to predict is "rent".

My dataset looks a lot similar to what happens in this notebook. I tried to normalize the rent column and the area column using log transformation since both columns had a positive skewness. Here's the rent column and area column distribution before and after the log transformation.

Before:

enter image description here

enter image description here

After:

enter image description here enter image description here

I thought after these changes my regression models would improve and in fact they did, except for Linear Regression.

If I don't do any type of transformations the models underperform. When I only transform the rent column all models improve including Linear Regression, but when I transform the rent column and the area column Linear Regression has a terrible result with a MAPE of 2521729.47.

Not transforming area MAPE results:

enter image description here

Transforming area MAPE results:

enter image description here

Can anyone tell me what's probably happening or guide me through any type of testing or verifications to understand what's happening to linear regression? Am I wrong to transform those columns even if the models are improving?

Edit:

After testing the models by removing and adding columns, I found that linear regression goes crazy after I insert the neighborhood column (which contains 66 neighborhoods) and create dummy columns. When I create this dummy variables the number of columns goes to 77, while the dataset has only around 3000 rows.

My thoughts are that after transforming the column into dummy columns the data becomes very sparse and with too many features for only 3000 rows, and that's why Linear Regression has this bad performance and Lasso Regression doesn't. Besides that, I should probably still use the other models since they perform well after the changes.

Am I correct?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ The results are so much worse that I suspect a bug or convergence failure. Could you provide some code, maybe minimized (reduce the feature set while keeping this behavior)? $\endgroup$
    – Ben Reiniger ♦
    Apr 22, 2021 at 14:10
  • $\begingroup$ After you comment I did some testing adding and removing variables, please check my Edit and tell me what you think :) $\endgroup$
    – Caldass_
    Apr 23, 2021 at 1:37
  • 1
    $\begingroup$ I can understand that dummy-encoding the neighborhood in this situation could cause overfitting, and that lasso would therefore outperform ordinary linear regression. But that the issue only happens when log-transforming the area variable is still surprising, as is the scale to which the dummy variables hurt performance. $\endgroup$
    – Ben Reiniger ♦
    Apr 23, 2021 at 15:31
  • $\begingroup$ I see, any suggestion or thoughts or this issue? $\endgroup$
    – Caldass_
    Apr 23, 2021 at 22:59

1 Answer 1

Reset to default
0
$\begingroup$

Make sure you transformed back your predictions and actual values before calculating MAPE.

You can check which observations contributed the most to high MAPE. MAPE is very sensitive to prediction errors at small actual values. Most likely worst performing observations ("from MAPE perspective") are those with small actual values.

Depending on the goal of your analysis you could check other metrics as well (eg: MAE).

Sparsity: Yes, you might have a neighborhood category in your test set, that does not exist in your training set (or has only a few examples). In this case predictions for that category might be very bad. Though this does not explain why you don't have high MAPE when you don't transform area.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Maybe is because most of the other numerical columns are ordinal, for example, number of bathrooms, bedrooms, etc. And the fact that I was transforming only area could be affecting the model predictions. What do you think? $\endgroup$
    – Caldass_
    Apr 29, 2021 at 0:38
  • $\begingroup$ Yes, perhaps taking the log of area caused some unlucky observations to get very high Absolute Percentage Error. For me log(rent) ~ log(area) linear model makes more sense. It means "if you add 1% to area, estimate of rent will get +x%. log(rent) ~ area linear model means "if you add 1 m2 to area, rent estimate will get +x%". I guess when 100 m2 increases to 200 m2 rent approximately doubles. But when 500 m2 increases to 600 m2, there's no double rent. $\endgroup$
    – gergelybat
    May 6, 2021 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.