I'm trying to estimate the benefit of using one method over another, but the existing data is confounded and I'm struggling to figure it out.

So my company submits debit orders to customers to collect a monthly payment. Sometimes customers reverse the charge or don't have enough money, so the payment fails and we don't collect any money from them that month.

Orders are submitted through method A, which is cheap but can be reversed; and through method B, which is more expensive but can't be reversed, although it can still fail. Overall method B is more likely to succeed in collecting a payment, and I'm trying to estimate how much better it is. How much more likely are we to collect a payment if method B is used over method A?

Currently a simple rule based system is used to decide whether to use method A / B for the next payment, which is based off of how many payments the customer previously failed. So the worse customers are more likely to have had their payments made on method B.

So when I look at my payments data, the average payment rate for method A is say 90% and the average payment rate for method B is say 70%. That makes sense, since the customers who were more likely to not pay are assigned to method B. But we know that method B is better at collecting payments!

So my issue is, how do I quantify how much better method B is from this data?


1 Answer 1


The only way is to obtain two comparable datasets for method A and method B. How exactly to do that is specific to your data and likely involves expert knowledge, but I can at least try to sketch possible directions:

  • If there are cases where customers were randomly assigned A or B and you have clear indications of these cases, that would perfect.
  • If there are enough cases where the same customer was sometimes billed by method A and sometimes B, that could the basic of a fair dataset: even if the customer was switched from A to B, one can still observe by how much this decreases the risk of non-payment. Note that the comparison would be paired by customer.
  • If none of this works, the company has to organize a proper A/B testing on a sample of transactions (basically building a dataset like described in the first option). This may cost a bit in the short term, but it could be beneficial to obtain reliable estimations in the long term.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.