0
$\begingroup$

I have a dataset containing a categorical feature with a missing rate 95%. What value can replace the missing cells? Or drop this feature?

$\endgroup$
  • $\begingroup$ If you do not think the feature is that important you could drop it. $\endgroup$ – Emre Apr 25 '17 at 18:24
  • $\begingroup$ Unfortunately, I know nothing about the feature whose name is a code and seems to be a secret. That is one of the reasons why I am so confused. $\endgroup$ – Noodle Apr 26 '17 at 2:10
0
$\begingroup$

You can turn it into a one-hot encoded feature with an added class of 'Missing', depending on the cardinality (how many categories are there). If the cardinality is too high, you will need to use other techniques for high cardinality features but you can still have 'Missing' as an additional category.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you.One-hot seems to be a great choice. And what would u do if the high missing rate feature is continuous variable? This is the very first time I have to deal with a dataset with about 490 features and 40000rows.And about 50% of the cells are 'missing' including continuous and categorical ones. Can't figure out what is the better way. It's driving me crazy...loll... $\endgroup$ – Noodle Apr 25 '17 at 12:51
  • $\begingroup$ Look for feature imputation, you could just use the mean of your training set of non-missing features or a more elaborate estimator, but I would add a binary column that shows if it was missing or not, because the fact that it was missing is sometimes information in itself. $\endgroup$ – Jan van der Vegt Apr 25 '17 at 12:53
  • $\begingroup$ yes, I was thinking missing may be a important information, too. While we add a binary column that shows if it was missing or not, we should fill the empty cells that is 'missing' as well. That's a feature imputation precess too, right? Will it raise too much noise? How does imputation (mean, EM...) affect the model, any paper I can consult? $\endgroup$ – Noodle Apr 25 '17 at 13:04
  • 1
    $\begingroup$ It really depends on what kind of model you use, but in my experience using something simple like the mean combined with the binary 'Missing' column works almost always as good as more elaborate models. I don't know any specific papers but there are plenty about feature imputation. $\endgroup$ – Jan van der Vegt Apr 25 '17 at 13:53
0
$\begingroup$

I have read the comments of another answer and seems like you have lots of missing data. I would then in this case recommend the mice imputation (multiple imputation with chained equation). It deals with all type of different variable types (numerical, categorical, binary) and fill the NA values depends on the type of the variable. If you use R, you can check https://cran.r-project.org/web/packages/mice/mice.pdf. It contains detailed package and function information and examples.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank u very much.what a pity, I didn't follow my tutor's advice learning R.But I will still have a check. BTW, where are you from? $\endgroup$ – Noodle Apr 26 '17 at 1:56
  • $\begingroup$ @Noodle I'm originally from China as you can probably guess from my name. $\endgroup$ – TYZ Apr 27 '17 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.