How can we handle cases were the quality of the prediction undermines the quality of the data?

Question

Consider the following example:

You have a rare disease whose occurrence seems to depend on a certain number of variables. You build a model which tries to predict patients most likely to be effected by the disease which is partially successful, that is, it predicts likely onset of the disease with some accuracy, but less than desired. Ideally, you update your model as more historical data on patients with this disease comes in, and the accuracy starts to improve.

Eventually you start notifying high risk patients and provide them with steps to counter this disease. Because of this, more and more patients who would have been classified as high risk are actually not catching the disease and therefore decreasing the accuracy of your model. In a sense the model was a 'victim' of its own success.

Are there any strategies for dealing with such prediction scenarios: Where a model designed to predict an undesirable outcome looses accuracy due to it successfully averting the outcome in real world cases?

Answer 1

Because of this, more and more patients who would have been classified as high risk are actually not catching the disease and therefore decreasing the accuracy of your model.

I'd rather challenge the assumption and say the algorithm didn't miss in this case: it correctly identified a high risk case, which was its original purpose. You're not building a model to identify people who will catch the disease, but to prevent the disease of actually happening.

Always think hard about which metrics make sense and what they represent. If you build a confusion matrix:

Predicted \ Actual | Caught the disease | Did not catch the disease
High risk          | A                  | B
Low risk           | C                  | D

Now, when you use accuracy, you're saying A and D are good, B and C are bad, all with the same weights. But is B really bad? Isn't C way worse than anything else? Or depending on the cost to prevent the disease, maybe B is worse? I don't know, it requires domain knowledge. But point being, using accuracy mindlessly is not a good way of dealing with this situation: adjust your metrics to your problem.

Answer 2

I suppose you can continue training your model after collecting new observations.

The simplest strategy would be to put more weight to the most recent observations (which are affected by model outcome).