0
$\begingroup$

(This is not a question on ways to handle missing data)

I have a dataset with around $80$ or so features and around $100000$ rows. Several of those features have missing (NULL) values for a "large proportion" (more than 20%) of the records.

Is there a general way to decide the % of the total rows for which a feature has to be missing, so that we can consider it useless and remove it from our analysis? Or does this depend on the context? Indirectly, I want to know the % of missing values for a feature so that imputation is meaningless.

| improve this question | |
$\endgroup$
  • $\begingroup$ You can just throw them away as such, fill them using sk-learn's imputer strategy seprately on train and test sets.... If you have a feature having nearly 90-95% missing values, then we (can) probably throw them away as People had done the same in Kaggle's Titanic Competition for the Drivers'Cabin $\endgroup$ – Aditya Apr 2 '18 at 14:47
1
$\begingroup$

In my opinion that depends on the context, like the question you want to answer, and also on the domain.

If the remaining absolute data set is still big and diverse enough to allow conclusions about that feature, the percentange of records that can not add to that imputation is not a problem, in my understanding.

It might also be that features sometimes are not just missing, but do not apply to the particular kind of record.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.