1
$\begingroup$

I am working on a fraud detection classification model with a dataset from a sample of labelled data. When the model is on production, we will be blocking transactions when the prediction of fraud is above a certain threshold. Hence, we will have no data for True Positives and False Positives. How can we evaluate the performance of the model once it is on production given that the labels will be missing? In the past, I have used by an approach that does not take action on a sample of datapoints.

Does anyone have any other ideas?

$\endgroup$

1 Answer 1

1
$\begingroup$

Did you ever find out some good information on this?

This is a similar problem in the Credit world (where loans are not given to applicants with credit score below a certain threshold), where it is referred to as Reject Inference, see for example https://www.mathworks.com/help/risk/reject-inference-for-credit-scorecards.html

In my experience, I have always relied on sampling to make inferences on blocked users. I.e., for users we are actioning on (blocking), sample x% of those users to get labels, where x is small but enough for robust metric calculation. Without sampling, I'm not sure what you could do, but A/B testing of model in production would be necessary as backtesting will have severe limitations without any labels on the blocked users.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.