0
$\begingroup$

I've been working my way through the features of the Kaggle House Prices dataset (Note: this is a non-ranking entry, so this is just for exercises), and I've found a couple situations where there is a positive correlation between the feature and the house sale price, but only if the data exists. In one case (shown below) over 10% of the dataset had null (means "does not apply", not "missing", so I can't fill it in with imputation), but of the non-null values, a scatter plot showed a positive correlation. This looks like a conditionally useful feature and I'd like to keep it, but the nulls are tripping me up.

I experimented and replaced the null values with 0, and when I looked at the scatter plot again, I found that the values that the new 0s spanned the a good chunk of the price range and altered the trend.

Before replacing null values (blue trend line estimated by hand): Before

After replacing null values with 0 (~13% of the dataset): enter image description here

Is there a readily available model in sklearn or some other python library that will perform a regression fit only if the data is non-null? If not, would it be best to just drop the column?

Note: This is a small dataset (<1500 entries). This is too small for neural network techniques.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

It is totally ok to drop null values here (in your case all the null values of LotFrontage), because this data isn't real. This is a standard part of data preprocessing called data cleaning. If the reason for the null values is known you could do additional steps like imputing or filling, but without this information just dropping them would be fine.

I would be careful changing null to zero however, as this is now creating data (i.e. apartments with zero size front lots).

Finally, you can see from your scatter plot that the relationship is not completely linear, so the linear model may have a large degree of error and/or low R squared values. So it may be more suitable to pick another algorithm for this dataset (e.g. 2D gaussian model).

Improve this answer
$\endgroup$
3
  • $\begingroup$ What do you mean by "drop null values"? I know how to drop an entire column, but I don't want to drop the rows because the dataset is already small. Also Kaggle doesn't allow dropped rows in its test datasets, so I couldn't do that anyway. $\endgroup$
    – John Cox
    Commented Mar 9, 2021 at 13:28
  • $\begingroup$ I meant in the second case, dropping the values of zero on the left side of the plot. This would be dropping rows, e.g. df.dropna() at least for this regression fit above. You can keep those rows for other analysis if they have other non-missing variables you want to use e.g. : df_cleaned = df.copy().dropna() $\endgroup$
    – WBM
    Commented Mar 9, 2021 at 13:45
  • $\begingroup$ Oh, gotcha. Well, a quirk about Kaggle (in my experience so far) is that the submissions expect the number of rows that were originally given in the test set, and I have to give the same treatment to the test set as the train set, so I can't just drop the rows. Thanks for the note about the 2D gaussian model though. I'll try that one. $\endgroup$
    – John Cox
    Commented Mar 10, 2021 at 5:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.