# Model Guardrails

Suppose I am building a machine learning model for an application where I do not need to make a prediction on all new samples, and given a new sample, it is better to make no prediction at all when there is concern that the prediction is unlikely to be good (for example, if the new sample appears to be very different than the training samples). I'm calling the idea of restricting which new samples to make a prediction on "model guardrails" because I don't know of an official term.

My question is, are there any standard methods of putting such guardrails in place? Is there any research on this topic that you can direct me to? A few basic ideas I have are:

1. Use a distance metric to compare the new sample to the training data and only make a prediction if there is a sufficient amount of training data sufficiently close to the new data.

2. Try to calculate some sort of p-value that indicates how dissimilar the new sample is to training data, and only make a prediction when this p-value is not too high.

To extend on idea 2), the exact method might have to depend on the training distribution, but in a simple case, perhaps one could calculate a p-value that represents the probability that sampling from the training data would yield a sample at least as far from average as the new sample (e.g. if we are regressing $$y$$ on $$x$$ and the training data appears to be a standard normal distribution, only make predictions when $$x$$ is within $$[-2, 2]$$).

I would appreciate references to the literature, a description of any standard techniques, or even just the right terminology to use.

• What you might be looking for is "uncertainty estimation". You want to know how confident your model is in its prediction, then keep only the high-confidence predictions. Some time-series models have confidence estimates built-in (like FB Prophet), and I've recently seen some work for estimating the uncertainty of neural nets: paper Uber blog post – zachdj Dec 5 '19 at 14:44
• @zachdj thanks for the references. Prediction intervals are somewhat related but not really the same thing I'm asking about here, because prediction interval accuracy generally still depends on the model having seen similar data, whereas I'm interested in identifying situations where the new samples differ too much from the training data to trust the model. – Seth Dec 5 '19 at 17:00
• In that case, perhaps anomaly detection models would be suitable. One cool idea is to train an autoencoder on your available data, then measure the reconstruction error of incoming samples. High reconstruction error indicates that the new sample is unlike those seen during training. Here are a couple of papers to check out: 1 2 – zachdj Dec 5 '19 at 17:34
• It's also possible to build Bayesian models which can essentially say "I don't know" when they are not confident in the prediction. I've mostly seen this with Bayesian Neural Nets But I imagine it would work similarly for other Bayesian models. – zachdj Dec 5 '19 at 18:28
• @zachdj "anomaly detection" seems to be what I'm looking for (and indeed the second paper you linked to mentioned as techniques some of the ideas I'd had, like distance based, cluster based, and kde based approaches). I will look at these papers more closely and also do some more searching using this term. Thank you! – Seth Dec 5 '19 at 19:00