0
$\begingroup$

Is it worth it to study missingness correlation between columns? If you have strongly correlated missing values (say between two columns, A and B), how will this change or shape the way you look at data? Does it add new information when analyzing data? Or, influence the analysis you would perform on it?

$\endgroup$

2 Answers 2

0
$\begingroup$

Usually the term correlation is used in context of association (and very often linear association). e.g. the daily price of stock A is highly correlated with the SP500 index. I am guessing here that you are thinking of occurrences of missings. e.g. How often when column A is missing is column B missing? Or When column A is missing does column B have specific values. And bring in other columns, when A and B are missing, then C is always > 100.

My answer is yes. Understanding the data, building more interactions into the data, going back and getting better data or questions why about the data are always good things. If 90% of the time that col A is missing then col B is missing also, this is a good question to look at how the data was acquired. Maybe use or not use the data. Maybe build indicator variables or interact the 2 columns, etc.

The more effort you put into the data, the better chance that the model has to be successful. It is good you are thinking about how to get creative analyzing your data.

$\endgroup$
0
$\begingroup$

Yes, studying missingness correlation between columns is valuable. It can reveal systematic reasons for missing data, suggest hidden relationships between variables, guide strategies for imputing missing values, and indicate potential bias in the dataset. This information can influence your analysis and preprocessing steps before modeling.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.