1
$\begingroup$

I am working on a credit risk binary classification problem. The classes are GoodPayers and BadPayers. The training set has variables/features that contains:

  1. DemoGraphics Data such as - Age, Education, Loan Amount, Interest Rate
  2. Behavioral Data such as - Payment in Month1, Payment in Month2, Payment in Month3, Payment delay in Month1, Payment delay in Month2.

The 10-fold cross-validation has 0.82 AUC on this set.

However, the unseen data just contains the 'Demographics Data' and does not have 'Behavioral Data' of Payment. How do we deploy/test the model based on DemoGraphics Dataset only?

$\endgroup$
2
  • $\begingroup$ Could you tell more about behavioral data? Would it be possible to train new models without these features? Are these features very related to your model performance? $\endgroup$
    – Theudbald
    Dec 23, 2017 at 17:32
  • $\begingroup$ @Theudbald link shows that the behavioral data are significantly important for achieving the model performance. Without these, the ROC drops to 0.65. $\endgroup$
    – vkb
    Dec 23, 2017 at 17:49

1 Answer 1

0
$\begingroup$

If less than 20-25% of behavioral data is missing, maybe you could try to impute missing data using one of the following solutions :

  • Impute missing behavioral data using some business rule or by training a machine learning model with demographics data as input and behavioral data as output variable.
  • Impute missing data with feature mean/median.
  • Impute missing data by picking up random value in the feature distribution (hot-deck).

In case you have more than 20-25% missing data, it will be really hard to impute values. I think in this situation you should consider creating a new model such as :

  • The new model doesn't use behavioral data anymore.
  • The new model is based on a different train-val-test split in order to have behavioral data in each dataset.

If you can't create a new model neither impute missing data, I guess hot-deck would be the best option you have to avoid bad performance on unseen data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.