Scikit Learn Missing Data - Categorical values

Question

I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier)

Label  Feat1 Feat2 Feat3 Feat4
 1      1     1      2     2
 2      3     1      2     2 
 3      4     3      3     1     
 4      4     1      2     4

I'm using scikit learn, and am considering using Naive Bayes or a Decision Tree. The classifier needs to be able to deal with missing features, and I read on scikit learn's page that Decision Tree does not support missing values.

What I am looking for is advice as to how to approach missing categorical values when using scikit learn. Also, any links to academic papers addressing this would be appreciated.

Answer 1

We would need more information on the prediction problem and the features to be able to give something more precise.

Anyhow, I am surprised no answer so far included all possible options since they aren't that many:

1. get rid of incomplete observations or features --- obviously, only viable if there are few incomplete cases since you lose too much information otherwise
2. replace NAs with some value like -1 --- this depends on the classifier you use; if your classifier supports categorical variables, you can create a new category for those NAs for example. In some continuous variables, sometimes there are some values that make sense (for instance, in text mining classification, if you have a title-length feature but you have no title, it might make sense to replace with title-length=0)
3. fill up the missing data

This last point encompasses too many things:

4. replace NAs with the median (this is the usual lazy approach; sklearn has a class for this)
5. if time series, replace with an average of the previous and following values -- in pandas, this can be done using DataFrame.resample().
6. use the $k$ closest neighbors. build a KNN model using the other variables and then do the average of those neighbors (if you use euclidean distance, you probably should normalize first). I never seen this done, but you probably could try predicting the missing NAs using another model as well.

But all this depends very much on what you are doing. For instance, if you have performed clustering analysis and you know your data is made up of clusters, you could use the median within each cluster.

Possibly other solutions could include things like multimodal or multiview models. These are recent techniques that can cope with missing modalities, and you can see a feature, or subset of features, as a modality. For instance, you could build a different classifier for various subsets of your features (using the complete cases in each of those subsets) and then build another classifier on top of that to merge those probabilities. I would only try these techniques if most of your data is missing. There are more advanced deep learning versions of this using autoencoders.

Answer 2

In my opinion it's always better to deal yourself with missing data instead on relying on classifier.

There are many ways to deal with it:

• Drop missing observations
• Drop rows where all cells in that row is NA
• Fill in missing data with any one of random label
• Fill label with maximum frequency for missing attribute

PS: I suggest using python pandas library for the purpose of data cleanup.

Answer 3

Instead of filling missing categorical value with median, I would use the mode. Doing this, you are sure to fill with existing modality even if you modalities are strings. Depending on the process behind the missing value, as said Ricardo Cruz, you could also add a new modality for the missing one

Answer 4

It's always a good practice to perform data cleansing before actually building a model and applying some algorithm on it. In order to data cleansing like handling missing values, "pandas" library is highly preferred. Here is the link to the "pandas" latest version, and that for "Working with missing values" reference in pandas.