I am working on a fraud detection model that prevents fraudulent users from using our solution. My model is performing great but the issue I have is that the more the model becomes performant the less I have fraudulent users in my training set and hence it becomes unbalanced compared with real world data. To cope with this, we have introduced a random process that lets some users pass without being scored so that we can keep learning from unbiased data. Ideally I should train my model on this unbiased dataset only, but it is small and it's a shame not to use the big part of the data. Hence, I would like to do the following:
- Train my model on the whole set : scored dataset (big but biased toward good users) + unscored dataset (small but unbiased)
- Calibrate the probability of the model using only the unscored dataset
What do you think of this ? Can you think of any drawback or bias it would introduce ?