I have a dataset containing a categorical feature with a missing rate 95%. What value can replace the missing cells? Or drop this feature?
2
-
$\begingroup$ If you do not think the feature is that important you could drop it. $\endgroup$– EmreApr 25, 2017 at 18:24
-
$\begingroup$ Unfortunately, I know nothing about the feature whose name is a code and seems to be a secret. That is one of the reasons why I am so confused. $\endgroup$– NoodleApr 26, 2017 at 2:10
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
You can turn it into a one-hot encoded feature with an added class of 'Missing', depending on the cardinality (how many categories are there). If the cardinality is too high, you will need to use other techniques for high cardinality features but you can still have 'Missing' as an additional category.
answered Apr 25, 2017 at 9:54
-
$\begingroup$ Thank you.One-hot seems to be a great choice. And what would u do if the high missing rate feature is continuous variable? This is the very first time I have to deal with a dataset with about 490 features and 40000rows.And about 50% of the cells are 'missing' including continuous and categorical ones. Can't figure out what is the better way. It's driving me crazy...loll... $\endgroup$– NoodleApr 25, 2017 at 12:51
-
$\begingroup$ Look for feature imputation, you could just use the mean of your training set of non-missing features or a more elaborate estimator, but I would add a binary column that shows if it was missing or not, because the fact that it was missing is sometimes information in itself. $\endgroup$ Apr 25, 2017 at 12:53
-
$\begingroup$ yes, I was thinking missing may be a important information, too. While we add a binary column that shows if it was missing or not, we should fill the empty cells that is 'missing' as well. That's a feature imputation precess too, right? Will it raise too much noise? How does imputation (mean, EM...) affect the model, any paper I can consult? $\endgroup$– NoodleApr 25, 2017 at 13:04
-
1$\begingroup$ It really depends on what kind of model you use, but in my experience using something simple like the mean combined with the binary 'Missing' column works almost always as good as more elaborate models. I don't know any specific papers but there are plenty about feature imputation. $\endgroup$ Apr 25, 2017 at 13:53
$\begingroup$
$\endgroup$
2
I have read the comments of another answer and seems like you have lots of missing data. I would then in this case recommend the mice imputation (multiple imputation with chained equation). It deals with all type of different variable types (numerical, categorical, binary) and fill the NA values depends on the type of the variable. If you use R, you can check https://cran.r-project.org/web/packages/mice/mice.pdf. It contains detailed package and function information and examples.
-
$\begingroup$ Thank u very much.what a pity, I didn't follow my tutor's advice learning R.But I will still have a check. BTW, where are you from? $\endgroup$– NoodleApr 26, 2017 at 1:56
-
$\begingroup$ @Noodle I'm originally from China as you can probably guess from my name. $\endgroup$– TYZApr 27, 2017 at 13:06