I am working on a classification problem. The dataset dimension is as 187,643 x 203. The first column contains class labels with no NA. The rest of dataset are frequency data and could be anything between 0 and 1. Here is a snapshot of the dataset
|class|groupA|groupB|groupC| ---------------------------- |0 | NA | 0.45 |0.001 | ---------------------------- |1 |0.001 |0.0008|0.001 |
The dataset contains high number of NAs. The min and max number of NAs in columns is as 24% and 90%, respectively.
To deal with NAs, I was thinking of defining a cut-off (let say 30%) for NAs count, dropping columns with NAs count greater than the cut-0ff value and replacing NAs in each remaining columns with class specific mean. However by doing this I will lose some features that seems to be very important for this classification job based on the data exploratory analysis. As an example, there is a variable with 64% missing value, but it can classify samples with AUC [95% CI]: 73.23 [72.86 - 73.61]. I do want to keep this kind of variables.
On the other hand, keeping missing values will require to use algorithms that can handle them like k-NN and Naive Bayes and random forest. But it seems sklearn implementation of these algorithms do not support presence of missing values.
update on 2021-08-25: I understand that there are recommendations to avoid discretizing a numerical features for most of ML tasks, but in this case if I convert the feature into a categorical variable and then assign a group name to NAs cases, I will be able to keep the feature. How does this sounds to you?
I am a bit new to this field and trying to figure out the best way to deal with NAs in the dataset.
I would appreciate any thoughts that you might have on this.