Dealing with high number of NAs in a classification problem

Question

I am working on a classification problem. The dataset dimension is as 187,643 x 203. The first column contains class labels with no NA. The rest of dataset are frequency data and could be anything between 0 and 1. Here is a snapshot of the dataset

|class|groupA|groupB|groupC|
----------------------------
|0    |  NA  | 0.45 |0.001 |
----------------------------
|1    |0.001 |0.0008|0.001 |

The dataset contains high number of NAs. The min and max number of NAs in columns is as 24% and 90%, respectively.

To deal with NAs, I was thinking of defining a cut-off (let say 30%) for NAs count, dropping columns with NAs count greater than the cut-0ff value and replacing NAs in each remaining columns with class specific mean. However by doing this I will lose some features that seems to be very important for this classification job based on the data exploratory analysis. As an example, there is a variable with 64% missing value, but it can classify samples with AUC [95% CI]: 73.23 [72.86 - 73.61]. I do want to keep this kind of variables.

On the other hand, keeping missing values will require to use algorithms that can handle them like k-NN and Naive Bayes and random forest. But it seems sklearn implementation of these algorithms do not support presence of missing values.

update on 2021-08-25: I understand that there are recommendations to avoid discretizing a numerical features for most of ML tasks, but in this case if I convert the feature into a categorical variable and then assign a group name to NAs cases, I will be able to keep the feature. How does this sounds to you?

I am a bit new to this field and trying to figure out the best way to deal with NAs in the dataset.

I would appreciate any thoughts that you might have on this.

Answer 1

There is no perfect answer about dealing with NA values:

• Sometimes it makes sense to completely remove a feature which has a high proportion of NAs (from all the instances). But in your case there are only 3 features, so this would lead to a huge loss of information.
• Imputing the missing values with the mean of the feature is another option indeed, but this artificial data can introduce bias in the model, especially if it's a high proportion of the data.
• Yet another option: remove the instances which contain at least one NA values. Since you have a very large amount of data, this could be considered in your case. But mind that this can also introduce a bias, especially if the distribution of the NA values is not random.
• An option which is often used with categorical features is to assign a special value NA. With numerical variables a similar approach would be to add a binary feature which indicates whether the instances has missing values (possibly one binary feature for every numerical feature). This might work with some learning algorithms but not with others.

There are some ML libraries which can handle missing values, I know that Weka does.

Answer 2

For any kind of preprocessing/feature selection process, domain knowledge is the primary and most important tool that one can have. So even if your model or the test you have done on the data says you should remove certain data, you should still keep it. Having said that, in your case you try some of the following things, in the same order:-

1.) If a certain feature has large number of NaN values (like say 90% of the values in a certain column is missing) it would be wise to drop that feature as imputing the values in that column won't make sense.

2.) If a column has high percent of NaN values but not like 90% (like in your case one feature has 65% NaN values), instead of imputing with mean or median, you could introduce a new column for that feature which gives the information as to weather that column contains NaN values or not i.e a binary value column. One advantage of this technique is that it will increase the information/data of your dataset, if in case you have extremely low number of features. But this method will only work with some algorithms as not all algorithms can handle missing values. (only Gradient Boosting algorithms can handle missing values as far as I know)

3.) This is the simple way out! Just impute the NaN values with either mean or median for numerical features and with mode for categorical features. (Remember to use mean when the difference between mean and median is low and use median when the difference is large).