# How to handle sparsely coded features in a dataframe

I have a dataset that contains information regarding diabetes patients, like so:

    id   diabetes   diet   insulin   lifestyle
0    No         NaN    NaN       NaN
1    Yes        Yes    Yes       NaN
2    No         NaN    NaN       NaN
3    Yes        NaN    NaN       NaN
4    Yes        Yes    NaN       Yes
5    Yes        Yes    Yes       Yes


Features diet, insulin and lifestyle have a high percentage of missing data (around 95% each). So initially, I excluded these features from my dataset. However, after taking a closer look at the data, I found the values for diet, insulin and lifestyle to be associated with the value for diabetes feature. This makes sense as diabetes patients would be recommended treatment relating to diet, insulin intake and lifestyle changes.

• So in cases where diabetes='No', values for features diet, insulin and lifestyle are missing.

• And in cases where diabetes='Yes', I have found that in most cases, at least one feature from diet, insulin and lifestyle to have a value of 'Yes', and the remaining values are missing.

After some reading, I believe features diet, insulin and lifestyle are Missing at Random (MAR), and clearly not missing completely at Random (MCAR) as is explained here.

Anyway, so my question is, should the nature of the missing data here change my decision to remove these features from the dataset due to their high percentage of missing values. Or, should I impute the data for these features, by filling in missing values with "No", like so:

    imputer = SimpleImputer(strategy='constant', fill_value='No')
x[:, 2:5] = imputer.fit_transform(x[:, 2:5])


The first question you have to answer is whether these are actually missing or simply sparsely coded!

E.g. if the variables are "supposed" to only show doctors recommended diet, insulin or lifestyle change then we could naturally conclude that any NaN is actually a "No" in which case you do not even have to impute the data but rather straight-up replace with "yes = 1 and NaN = 0.

How do you differentiate this case (sparsely coded) from the other case: actual missing values? Besides applying your domain knowledge you should also test whether the following rules apply:

• If diabetes == No then all treatment variables are missing
• If diabtes == Yes then at least one treatment variable is "Yes"
• Treatment variables are always only "Yes" or NaN

If this is the case your dataset is likely sparsely coded, otherwise you have actual missing values.

Take note however that should the values be actual missing values (e.g. because you identify some cases where non-diabetics have a treatment or some cases are labeled "No" in the treatment variables) you can assume they are MAR but not MNAR. In this case I would recommend to remove these variables because imputation as "No" or o does not make sense if you do not have any information what the NaN actually means.

• Thank you for this detailed answer. What if part of these rules apply and others do not? So checking the dataset again, the treatment variables are always only "Yes" or NaN, but there are some cases where diabetes == Yes but all treatment variables are NaN. Could I just assume that some patients who were diagnosed with diabetes did not accept any treatment options? Jul 24, 2020 at 9:37
• @sums22 this seems likely to me but is of course impossible to know without knowing how the data was collected. Pragmatically I think it is fine to go ahead with this assumption but keeping in mind that a bad performance in validation maybe due to this. With prediction tasks we have the luxury of only caring about performance so if the model performs well in validation all is good. Jul 24, 2020 at 9:40
• Makes sense, I will go ahead with it and see what results I get. Many thanks for the help. Jul 24, 2020 at 9:50