I have recently discovered Sklearn's MissingIndicator but still wondering how could it improve the usual machine learning work. Clear that

from sklearn.impute import MissingIndicator
indicator = MissingIndicator()
X_mask = indicator.fit_transform(X)

could result in X_mask as boolean indicator array of missing values in original X. But how could it be utilized actually beyond that it keeps track where imputed values are? I know that flagging NAs in a separate column per features could be useful for learners but here X_mask is not combined with the imputed X so this is seemingly not the point.

Similar question arises with putting MissingIndicator in the ML pipeline:

transformer = FeatureUnion(
         ('features', SimpleImputer(strategy='mean')),
         ('indicators', MissingIndicator())])

What is the point to add the pure mask array to the imputed array here?


Good question. Here is my take on it:

The following piece code creates the union of both the imputed features and the indication of which ones were imputed.

transformer = FeatureUnion(
         ('features', SimpleImputer(strategy='mean')),
         ('indicators', MissingIndicator())])

I can see it becoming useful for a learner that may learn to not rely as much on features that are often missing. Or perhaps, the fact that some features are missing at all can be helpful for the learner. As, missing values can mean that something happened differently in the data collection, which can be useful to make a prediction.

Let's say I have the following problem:

I want to classify users based on their watching habits on my fancy streaming platform. Users have to rate a movie thumbs up or thumbs down. Of course, not all my users will watch all movies, so I'll have blank values. By using let's say the SimpleImputer, I'll basically make an educated guess on how they would have liked the movies they didn't watch. That can be useful as then I can compare users across all movies. However, knowing which movies the user has watched altogether is probably as informative. This is what the MissingIndicator provides you.

  • $\begingroup$ OK, thanks, I see the logic, but what actually is the "union of both the imputed features and the indication of which ones were imputed". How does it look like? It must be some matrix that could be feed to a learner, mustn't it? $\endgroup$
    – Fredrik
    May 9 '20 at 6:22
  • $\begingroup$ It's simpler than you think. FeatureUnion combines multiple transformers/preprocessing steps and then concatenates its results. That way, you get the "union" of all features $\endgroup$ May 9 '20 at 11:00

The idea of tracking missing data can be useful, so you know which data-points were imputed or dropped for training a model. It can give you an idea of feature importance/reliability.

However, there are many way to do this and if your are using e.g. a Pandas DataFrame, the MissingIndicator class is redundant for the single purpose of tracking missing values in my opinion.

The same can be achieved as follows. Say I start with this dataframe:

import pandas as pd
import numpy as np

In [1]: df                                                                                                           
            A         B          C         D
0    1.095564 -0.225533   0.441428  0.099792
1    1.198053  0.523837   -1.53928  0.871108
2         NaN  0.336165  NOT_FOUND -1.881777
3   -0.077794  0.175203   -1.76324  1.172351
4   -1.167858  0.340200   0.369765       NaN
5    0.514393 -0.045929   0.771916  0.130821
6   -0.065623  0.978825  -0.668706 -0.703892

So there are a few possible missing values:

In [2] missing_vals = ["NOT_FOUND", np.NaN]

In [3] missing_mask = df.isin(missing_vals)   # boolean mask of True where missing values found

In [4]: missing_mask                                                                                                 
       A      B      C      D
0  False  False  False  False
1  False  False  False  False
2   True  False   True  False
3  False  False  False  False
4  False  False  False   True
5  False  False  False  False
6  False  False  False  False

But as this is a DataFrame, you have all the power of that if you need it. See how many values are missing per feature:

In [5]: missing_mask.sum()
A    1
B    0
C    1
D    1
dtype: int64

Replace all missing values with a single marker e.g. "MISSING"

In [6]: df.where(~missing_mask, "MISSING")
           A          B         C          D
0    1.09556  -0.225533  0.441428  0.0997919
1    1.19805   0.523837  -1.53928   0.871108
2    MISSING   0.336165   MISSING   -1.88178
3 -0.0777938   0.175203  -1.76324    1.17235
4   -1.16786     0.3402  0.369765    MISSING
5   0.514393 -0.0459287  0.771916   0.130821
6 -0.0656233   0.978825 -0.668706  -0.703892

If you want to understand your missing data, I suggest having a look at this great package called missingno, which was built for that purpose :)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.