Let feature $feat$ contain one value $A$ that occurs 5% of the time, while 95% of the time it is empty.

Instead of arbitrary saying features that have less than 5% should not be included into the model, is there a way to quantify mathematically this percentage?


Why do people say that?

when you have that much missing values (95%) than that could be encoded as special value, and it will look quantitatively as a value in itself. Now when you start using this feature as "usable" what will most likely happen is that you get noise, i.e. conflicting information/non-discriminatory one. In other words this feature will be saying the same information for all the classes (since there is 95% of it, its most likely). Hence you can do this encoding, fit a random forest and look at gini coefficients (for example!) and you quantify mathematically how important this feature is relative to the problem and the dataset.

If you are looking for quantifying mathematically this percentage generally, thats impossible since its all relative and problem specific. From experience 50%+ missing values is extremely hard to work with.

CAVEAT you dont even have to encode it as value in itself, just fit lgbm model, there missing values will be assigned with values that minimises the loss, and you may even find out that its not that useless after all.

  • 1
    $\begingroup$ Interesting.. can we use lime package to know feature importance here? $\endgroup$
    – cap
    Dec 20 '19 at 23:01
  • $\begingroup$ sure can, thats why I said "for example!" $\endgroup$
    – Noah Weber
    Dec 20 '19 at 23:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.