Data imputation for heavily missing features

Question

I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, to the point where the vast majority of the features are unavailable. The dataset has transformed 300 of its features into Principal Components for privacy protection, and therefore, it is unable to understand the meaning of the feature.
I am wondering what are some good approaches to this kind of problem. I have thought of data imputation, but making up and filling in fake data for the majority of the feature seems unrepresentative of real life data. I have also thought of dropping the features with many non-values, but that just seems like disregarding information, which could prove to be crucial.
I would appreciate any response, and thank you very much!

Answer 1

The best way for imputation depends not only on tha data, but also on the models that operate on the data. Not knowing the data and not knowing model, all I can do is to give some approaches.

• You could perform a normal imputation and add for each column with nans another binary column that indicates whether the value was imputed or originally there. By doing so, the information is not lost to the model and some models might learn to use these binary flags.
• There are some models that can handle nan-values. Some tree based models like XGBoost store for example a default child for nan-values in each node.
• There are more complex imputation methods that you could try. One could for example try to predict a nan-value based on the values of the other columns. This might be computational expensive and take some interations to converge (if the prediction of a value in column A is based on nan-values in column B and afterwards the value in column B is imputed in a similar way, then one might want to update the imputation in column A as well).

Answer 2

If you don't want to have a big data loss you can use the function fillna from the Pandas library. It uses its "method" parameter.

method: {‘backfill’, ‘bfill’, ‘ffill’, None},default None

Method to use for filling holes in reindexed Series:

• ffill: propagate last valid observation forward to next valid.

• backfill / bfill: use next valid observation to fill gap.

example:

df = pd.DataFrame([[np.nan, 2, np.nan, 0],
                   [3, 4, np.nan, 1],
                   [np.nan, np.nan, np.nan, np.nan],
                   [np.nan, 3, np.nan, 4]],
                  columns=list("ABCD"))
df
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  NaN  NaN NaN  NaN
3  NaN  3.0 NaN  4.0

df.fillna(method="ffill")
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  3.0  4.0 NaN  1.0
3  3.0  3.0 NaN  4.0