0
$\begingroup$

So I have this dataset from census the census bureau database which contains 40 attributes and a target column specifying if the total income is >50k. P.S : this is not the known ADULT dataset (which is a lot cleaner), the difference is that mine wants to predict the total person income rather than the adjusted gross income.

My problem is that some of the attributes such as : class of worker (meaning if he's self employed, local government...), major occupation and and industry codes have 100 000 missing values knowing that the total length of my dataset is 150 000.

Now it bugs me to simply drop those column because it seems to me like an important information to know if we want to predict the total income.

What should I do in such a position ?

Thanks !

$\endgroup$

closed as unclear what you're asking by Stephen Rauch, Sean Owen Oct 19 '17 at 15:15

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

2
$\begingroup$

I would first build some models without those predictors. If 2/3 of your training set is missing so you can't have your basic model based on them. If your accuracy is too low I would try to use a smaller data set and not use the missing values. Those kind of missing values are not somthing that you can easily define (like a default job) unless you have a prior on your data (e.g most people in the data are data scientists). If you do have any kind of such prior you can use missing data methods more efficiently.

$\endgroup$
0
$\begingroup$

I would say before drop the columns if could verify the entropy (see here) of each one to see if they are important feature to your process or not. This idea is more common in classification problems, but maybe you can do some adjustments on yours.

For numeric features you can try work with the simple interpolation solution (see here).

Finally I like a book from Dr Graham about how to handle missing data called Missing Data Analysis and Design.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.