3
$\begingroup$

I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k records in the dependent variable that have missing values. - How do I solve this problem? Should I be excluding these records? - Is imputing the missing values a good method of solving this problem? Wouldn't that give me inaccurate values as the dependent variable is dependent on a number of variables? I'm not sure.

Can anyone please help?

Improve this question
$\endgroup$
3
  • $\begingroup$ Isn't the task to predict the dependent variable? If the dependent variable is missing, you simply can't use that data. $\endgroup$
    – Harsh
    Commented Nov 11, 2018 at 2:48
  • $\begingroup$ I'm not sure if this is what you're asking for, but semi-supervised methods can make use of a large amount of unlabeled data and a small amount of supervised data to make a classifier. You seem to have very few such data points, so you might not find this worth it $\endgroup$
    – Harsh
    Commented Nov 11, 2018 at 2:50
  • $\begingroup$ The task is to apply any ML model which I find it apt for this problem statement. It could be literally anything - regression, classification or clustering. $\endgroup$
    – Aditya Kadrekar
    Commented Nov 11, 2018 at 4:49

1 Answer 1

Reset to default
1
$\begingroup$

The questions you're asking are empirical questions. The only answer anyone can give is to try all of them and see which works better.

You have three options:

  • Impute data

  • Throw away data

  • Use a classifier that can handle missing data, e.g. xgboost. See this answer. xgboost is a powerful classifier. So, if you're not tuning very hard for performance, xgboost is a great way to get a good v0.

Some other points:

  • The pattern of missing values is important, and can influence the choice of algorithm.

  • If your dataset is noisy, imputing may only amplify the noise. If you can afford to drop those 2k rows, try that, or train both with and without that data and see if the combination performs better.

Regarding software, there are many options:

  • Scikit Learn has some imputation functions

  • MICE, Multiple Imputation through Chained Equations works well for random data. Available in fancyimpute, and also in statsmodels.

You will find many resources if you search

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your inputs! I'm going to try two methods - 1. Exclude the missing data and then just split the dataset into train and test set 2. Keep all the missing data in test set. Train the model and predict the values. And then compare with the imputed mean. Basically, I have been given a dataset with a target variable. The concern here is there is no metadata given. So it's hard to understand which independent variables might influence the target variable. Year, month, day and hour are also given as columns. $\endgroup$
    – Aditya Kadrekar
    Commented Nov 11, 2018 at 4:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.