I am working on the KDD 2009 Cup Data Set (The Small one) and I have a question about preprocessing data. It has a lot of columns with null values, some of them have more than 90% of missing. Reading some papers on the challenge I noticed that even though these columns have this amount of null competitors usually did not excluded them. I was doing some validation and in order to argue that if I should drop it or not, I calculated the mean of target variable comparing null vs non-null, something like this: enter image description here

You can see for example, that for variable Var118, for churn and appetency targets they do not have a significant difference on mean target values. However, for up_selling, we can notice some discrepancy. My question is: is that a correct approach? I am having second thoughts because my data set is extremely unbalanced, and maybe my target means will be biased. Anyone have had some experience on this type of problem? I would love discuss more about that.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.