1
$\begingroup$

I'm preparing features for a neural network which I'll run in Keras and TensorFlow. The features are generated in Oracle. There I also replace null values. I'm not doing normalization on the database because this depends on the chosen sample. Normalization will be done in Python. For my use case, which is a binary classification problem (fraud detection), the presence of null values also correlates with the target variable. Therefore, I would like to preserve this information for the model.

My proposal would be to create an additional binary column called varName_isnull which encodes the presence of a Null in the varName column. In other aggregated features, this binary column would also be used for example to calculate the number of Null values for a particular grouping (i.e. credit card) per unit of time.

Is my proposal reasonable? Are there any alternative representations and if yes what would be their advantages?

| improve this question | | | | |
$\endgroup$
1
$\begingroup$

It would depend on the type of model you want to train. If you are doing a classification model ( Yes/No, 0/1 type of class) then I suggest you use dummy variable generation functions in python.

I would also recommend that you convert all your other categorical variables into dummy variables in order for your model to take every bit of information into account.

one popular technique in feature engineering is one-hot-encoding to change the nature of your data.

So generally you would create a new variables ( as you have mentionned) that has 2 values :

  • 1 if the variable is NULL
  • 0 if the variable has a non-Null value Below are some links with nice examples:

https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

https://jamesrledoux.com/code/dummies

Good luck !

| improve this answer | | | | |
$\endgroup$
0
$\begingroup$

The most obvious disadvantage to your idea is data size. If your dataset is small then this may not be a problem, but at scale potentially doubling the number of columns could be an issue.

One idea could be to encode the missing value with a value that is completely different from the "true" values - some examples:

  • With a continuous variable which has values in [0,100], encode the missing value as -1
  • With a categorical variable with level names "A", "B", "C", encode the missing value as 999 (or its own level).

I think if you're careful with your encoding you will be able to identify the missing values and so still be able to generate the group statistics you mention above.

| improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.