How to deal with possible data leakage in time series data?

Question

I have historical consumer data who have taken out a loan at some point in time. The task is to predict if a consumer will default when requesting a loan.

My issue is that for some customer in the data set, historical transactions are only available after the loan was issued. I believe using data after the loan event for prediction will cause data leakage.

This is a subtle leakage because it does not involve using information not available at prediction time. My concern is more about behavioral change when the customer is indebted that create a shift in the underlying distribution.

To test my hypothesis I was wondering if comparing whether the two samples before and after the loan is issued come from the same distribution will be a good approach.

These are my questions:

1. Is there really data leakage in the scenario I described

2. If yes, can I test it in any way?

3. Can a two samples test provide an answer? Which one? Note that the sample is composed of multivariate data

4. Can I do testing using any machine learning approach? I was thinking of using a Mixture Model to test for instance.

Any suggestion on how to best deal with this problem other than what I suggested will be appreciated.

Thanks

Answer 1

There is no need for sample tests.

A customer may have received many loans 1 to n - 1. To predict the default rate of nth request at time t(n), you are allowed to use any information up until t(n). When a user has no transaction history before t(1) system cannot predict the default rate for her; except maybe based on her age, income, etc. However, for the next loan request at t(2) system can use the transactions between t(1) and t(2), but still cannot use any transaction that happened after t(2). For any particular prediction at t(n), events happened after t(n) must never be used.

Regarding "it does not involve using information not available at prediction time", it does involve using information not available at prediction time t(n), since system is trying to utilize transactions that occur after t(n).