I'm working on a dataset of banking transactions and would like to find recurrent transactions. I've been mapping transactions per merchant in timeseries, and tried to use acf from statsmodels.tsa.stattools to calculate the autocorrelation function but i'm not getting the expected results:

r = acf(ts, fft=False)

For example this set of transaction (ASSURANCE DESJ) is getting an acf score of 0.3159 when it's obviously a recurring transaction (same amount, same frequency).

enter image description here

Another example of recurring transactions with acf=0.22775:

enter image description here

But this one should not be found as a recurring transactions, and get a score not too far from the previous set (0.26919):

enter image description here

I've been checking a lot of different methods, I acutally came up with a combination of auto-correlation on the regular timeserie, auto-correlation on a timeserie with amount=1, with stationnary checking and other rules to have a not so perfect results. I've also checked at ARIMA and other methodology without luck.

Would you have a better way to detect recurring transactions from timeseries ?

Link to Datasets


I don't know the model you use, but I would suggest the perhaps old-fashioned feature engineering approach:

  • In each set of transactions, calculate the duration between two consecutive transactions for every pair of consecutive transactions (i.e. difference between the dates). Then add the standard deviation of these durations as a a feature, so that a very regular transaction set should get a near zero value.
  • Same idea with the amount of the transaction: the standard deviation will be zero if the amount never changes.
  • Additional ideas:
    • number of distinct amounts across all transactions
    • number of distinct duration values between transactions in days, weeks, months, years.
    • standard deviation of the sum by day/week/month/year

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.