Data preprocessing

Question

I just want to know how to determine whether to remove the missing values or to impute them with mean , median or mode. I usually remove the missing values but it decreases the size of the dataset by more than 50%. So can anyone guide me in this regard. I am new in this feild.

Answer 1

• Ignoring a tuple is effective when it contains several attributes (>50%) with missing value OR the value of target variable (class label) is missing.
  
• In other cases, you can either impute them with mean, median or mode, OR you can fill them with most probable values (using interpolation or regression).

Answer 2

Target column:

• If you have missing data in target column, then it's better to remove those rows.

Numeric column:

• Usually it is good to impute missing data with the median data when the column is numeric.
• If the data is numeric in nature and the data is missing, then try to plot the distplot and see if the data is normally distributed or not. Now, if your data is normally distributed then you can use either mean or median as they will most probably will have very close vale.

Categorical column:

• Or you can impute the missing numeric values with the values like -999 and treat them like outliers and you either remove those outliers or use some transformation over it.
• Lastly, if your data is categorical in nature, then you can replace the missing values with values like -999. And then based on your data (whether it's ordinal or not) you can choose to either use one hot encoding or labelencoder.
• Also, for categorical data, you can impute the values with the mode value as well.

Answer 3

You should determine if the attribute is Missing at Random (MAR), Missing Completely at Random (MCAR), or Missing Not at Random.

For each of these categories there are recommendations when and how to impute.

But how to classify attributes in your dataset as MAR,MCAR,MNAR?
By doing EDA, e.g., studying the statistical distribution of the attribute values.

It is best to read modern academic literature about this topic (I cannot explain it properly and in full here).

You can also ask ChatGPT or any other LLM, e.g., with this prompt:how to determine if an attribute is Missing at random , Missing not at Random, Missing completely at random?

...It's important to note that determining whether missing data is MCAR, MAR, or MNAR is often difficult and requires careful consideration of the data and the research question. ...