Do models of social systems suffer from prediction drift?

Question

Background

I've created a binary classification model that predicts the probability of fraud for a given sample.

The choice of threshold allows me to set how many frauds are captured in the training dataset. However, when I test the model on a similar dataset that was collected a year later, the threshold must be lowered to capture the same number of samples.

Issue

This implies that my model is under-predicting the probability of fraud in the new dataset. This implies that the model's predictions are drifting towards 0 and that the distribution is decreasing and narrowing.

Question

Is this a normal occurrence in models that are built on complex social systems?

Answer 1

I wouldn't call it normal but it surely is possible. There are several reasons:

1. Changes in fraud patterns: If the model was trained on historical data that is no longer representative of current fraud patterns, it may not be able to detect the latest types of fraud.
2. Drift: data drift occurs when the statistical properties of the data being analyzed change over time. For example, if the demographics of the population being analyzed change, or if new types of transactions are introduced, the model may not be able to detect fraud as effectively.
3. Model Decay: The model can lose efficiency over time due to changes in the environment. It is similar to the drift. You have to retrain or refit the model using new data periodically to avoid this.
4. The fraudsters have hacked you, they now know how the model works and are avoiding detection :)

It is most likely a normal occurrence. I suggest refitting.