Dealing with unbalanced training set compared with real world data

Question

I am working on a fraud detection model that prevents fraudulent users from using our solution. My model is performing great but the issue I have is that the more the model becomes performant the less I have fraudulent users in my training set and hence it becomes unbalanced compared with real world data. To cope with this, we have introduced a random process that lets some users pass without being scored so that we can keep learning from unbiased data. Ideally I should train my model on this unbiased dataset only, but it is small and it's a shame not to use the big part of the data. Hence, I would like to do the following:

1. Train my model on the whole set : scored dataset (big but biased toward good users) + unscored dataset (small but unbiased)
2. Calibrate the probability of the model using only the unscored dataset

What do you think of this ? Can you think of any drawback or bias it would introduce ?

Answer 1

There are a few things you can do. Keep in mind there are some people who would disagree with one method while others agree on the same.

1.) Since it an imbalanced dataset, you can apply any of the sampling techniques (either oversampling or undersampling). I would suggest oversampling (SMOTE) since undersampling leads to information loss. Now there are people who disagree with this method and instead advise the second method.

2.) Change the metric you are using. You should always use metrics like weighted AUC, F1 score for imbalanced datasets.

There is no clear cut winner as to which technique is right. Sampling techniques have been used by many people but on the other hand, some experts say it only introduces noise into the data and changing the metric can solve the problem.

Answer 2

It is great you are thinking like this. Often when we build a model, we do not think how we will get the data for the next model build or the biases in this building model from previous models. In many models, like fraud or credit, the current model and business policy biases future data.

I have built fraud models with event rates < 1%. Personally I do not like smote type of techniques since they create data - who said that some random point between two other points should be categorized the same. I play with slight undersampling, often by weighting, to not lose too much information. Not always random undersampling but finding matches, clusters, and example transactions. And I do not always undersample.

I think the random process should be turned into a test - check out design of experiments. Then you test specifically from areas of the population to get good coverage. It is hard to test in fraud for business reasons, so like you said, your test data will be thin. You may also not want to test in certain areas that are too risky - like if your model is about financial fraud and you do not want to test where the loss may be > $10,000.

In your train/validation/holdout data sets, include some of the test data. Run specific metrics - whatever makes sense for your business problem - on the test and non-test data. Now you have some unbiased metrics and a potentially better view how the model will perform on newer data. If the test data is very thin, put it all into the holdout set.