Take two step approach.
- What should be imputed at missing data points?
- Can we add a new column to indicate missing second transaction?
E.g. First, impute the missing values with a negative value. Second, create additional
Col1_flag which will have binary value
Yes indicating missing second transaction- the reason for
No indicating normal scenario where the
Col1 has legit value.
Now, this you will need to do for all the
6 columns i.e. impute with a negative value for missing values and create a new feature for each such column to indicate condition of transaction. Therefore, you will have 6 new features also to be included in modeling. This work successfully, please try and let me know.
@Roger a detailed discussion below:
Here is a simple approach that works for filling
Null for the transactional data points which have large number of missing values due to nature of feature and one needs to keep such columns to ensure business logic flow into the machine learning model when training.
Col1 could be the average number of days between 2 consecutive transaction. If a customer only purchased once then the average is NULL.
First, the imputation for the missing values need to be numeric. And because there exists only one transaction, we cannot calculate difference, mean, median, or std. Therefore, in such scenario, impute the missing values in the
Col with a large negative value e.g.,
-9999. This will help send signal to machine learning model about special importance of missing value at the same time keeps the transaction entries intact. Thus, we don't compromise on number of data points.
||Missing value imputation