# Are there readily available models that can handle conditional correlation?

I've been working my way through the features of the Kaggle House Prices dataset (Note: this is a non-ranking entry, so this is just for exercises), and I've found a couple situations where there is a positive correlation between the feature and the house sale price, but only if the data exists. In one case (shown below) over 10% of the dataset had null (means "does not apply", not "missing", so I can't fill it in with imputation), but of the non-null values, a scatter plot showed a positive correlation. This looks like a conditionally useful feature and I'd like to keep it, but the nulls are tripping me up.

I experimented and replaced the null values with 0, and when I looked at the scatter plot again, I found that the values that the new 0s spanned the a good chunk of the price range and altered the trend.

Before replacing null values (blue trend line estimated by hand):

After replacing null values with 0 (~13% of the dataset):

Is there a readily available model in sklearn or some other python library that will perform a regression fit only if the data is non-null? If not, would it be best to just drop the column?

Note: This is a small dataset (<1500 entries). This is too small for neural network techniques.

It is totally ok to drop null values here (in your case all the null values of LotFrontage), because this data isn't real. This is a standard part of data preprocessing called data cleaning. If the reason for the null values is known you could do additional steps like imputing or filling, but without this information just dropping them would be fine.
Finally, you can see from your scatter plot that the relationship is not completely linear, so the linear model may have a large degree of error and/or low R squared values. So it may be more suitable to pick another algorithm for this dataset (e.g. 2D gaussian model).
• I meant in the second case, dropping the values of zero on the left side of the plot. This would be dropping rows, e.g. df.dropna() at least for this regression fit above. You can keep those rows for other analysis if they have other non-missing variables you want to use e.g. : df_cleaned = df.copy().dropna()
• Oh, gotcha. Well, a quirk about Kaggle (in my experience so far) is that the submissions expect the number of rows that were originally given in the test set, and I have to give the same treatment to the test set as the train set, so I can't just drop the rows. Thanks for the note about the 2D gaussian model though. I'll try that one. Mar 10, 2021 at 5:01