Null values as useful information for feature engineering

Question

I'm preparing features for a neural network which I'll run in Keras and TensorFlow. The features are generated in Oracle. There I also replace null values. I'm not doing normalization on the database because this depends on the chosen sample. Normalization will be done in Python. For my use case, which is a binary classification problem (fraud detection), the presence of null values also correlates with the target variable. Therefore, I would like to preserve this information for the model.

My proposal would be to create an additional binary column called varName_isnull which encodes the presence of a Null in the varName column. In other aggregated features, this binary column would also be used for example to calculate the number of Null values for a particular grouping (i.e. credit card) per unit of time.

Is my proposal reasonable? Are there any alternative representations and if yes what would be their advantages?

Answer 1

It would depend on the type of model you want to train. If you are doing a classification model ( Yes/No, 0/1 type of class) then I suggest you use dummy variable generation functions in python.

I would also recommend that you convert all your other categorical variables into dummy variables in order for your model to take every bit of information into account.

one popular technique in feature engineering is one-hot-encoding to change the nature of your data.

So generally you would create a new variables ( as you have mentionned) that has 2 values :

• 1 if the variable is NULL
• 0 if the variable has a non-Null value Below are some links with nice examples:

https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

https://jamesrledoux.com/code/dummies

Good luck !

Answer 2

The most obvious disadvantage to your idea is data size. If your dataset is small then this may not be a problem, but at scale potentially doubling the number of columns could be an issue.

One idea could be to encode the missing value with a value that is completely different from the "true" values - some examples:

• With a continuous variable which has values in [0,100], encode the missing value as -1
• With a categorical variable with level names "A", "B", "C", encode the missing value as 999 (or its own level).

I think if you're careful with your encoding you will be able to identify the missing values and so still be able to generate the group statistics you mention above.