Exploratory Data Analysis

Question

I am working on this dataset. Dataset has missing values. What would be the best method to impute the missing values. Also values are missing from target feature as well. So far I have dropped those observations from the dataset.

There are multiple instances of the same ID for which some variables will not change i.e. age, weight, height etc., but some values are missing. I am trying to impute the missing values by other given values for the same ID. How can I go about it in Python? Thanks in advance.

Answer 1

To deal with missing data you can use one of the following three options:

• If there are not many instances with missing values, you can just delete the ones with missing values.

• If you have many features and it is affordable to lose some information, delete the entire feature with missing values.

• The best method is to fill some value (mean, median) in place of missing value. You can calculate the mean of the rest of the training examples for that feature and fill all the missing values with the mean. This works out pretty well as the mean value stays in the distribution of your data.

Note: When you replace the missing values with the mean, calculate the mean only using training set. Also, store that value and use it to change the missing values in the test set also.

In python you can use the Imputer() class to fill up the missing values as follows:

from sklearn.preprocessing import Imputer
impute = Imputer(strategy="median")
impute.fit(df)

Also you can check the calculated median values for each column by using:

imputer.statistics_