0
$\begingroup$

I am working on this dataset. Dataset has missing values. What would be the best method to impute the missing values. Also values are missing from target feature as well. So far I have dropped those observations from the dataset.

There are multiple instances of the same ID for which some variables will not change i.e. age, weight, height etc., but some values are missing. I am trying to impute the missing values by other given values for the same ID. How can I go about it in Python? Thanks in advance.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Please give a correct link, or write the sample data. $\endgroup$
    – ipramusinto
    Oct 8, 2018 at 7:38
  • $\begingroup$ The answer depends on why you want to impute missing values. Are you trying to pipe this data into a machine learning model? Or to display back to users? Or something else? $\endgroup$
    – tom
    Oct 8, 2018 at 19:46
  • $\begingroup$ I want to put it into a machine learning model. $\endgroup$
    – akhilesh sayana
    Oct 9, 2018 at 11:40

1 Answer 1

Reset to default
1
$\begingroup$

To deal with missing data you can use one of the following three options:

  • If there are not many instances with missing values, you can just delete the ones with missing values.

  • If you have many features and it is affordable to lose some information, delete the entire feature with missing values.

  • The best method is to fill some value (mean, median) in place of missing value. You can calculate the mean of the rest of the training examples for that feature and fill all the missing values with the mean. This works out pretty well as the mean value stays in the distribution of your data.

Note: When you replace the missing values with the mean, calculate the mean only using training set. Also, store that value and use it to change the missing values in the test set also.

In python you can use the Imputer() class to fill up the missing values as follows:

from sklearn.preprocessing import Imputer
impute = Imputer(strategy="median")
impute.fit(df)

Also you can check the calculated median values for each column by using:

imputer.statistics_
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.