How to predict the quality (as a classification) of a regression?

Question

For an industrial workflow we created an ANN (in tensorflow) for a regression problem where we take customer inputs (as numeric values) and predict the measurements for the units that need to be produced (length, height, etc).

We analyzed the errors and figured out that basically 50% of the predictions would be within acceptable tolerances while in the other 50% a human expert should review them. The problem we have is: How to predict, if the regression is within tolerances or not?

Our first idea is to label our training-data as within_tolerance or outside_tolerance and build a classifier with this. Is that a common approach? If so, what is the term for this kind of process so that we could look it up in the literature?

Answer 1

In general this idea isn't bad BUT you need to collect new data for this to work.

Modeling isn't suspect to p-hacking but this would be quite similiar. 50% of predicted outcomes not within tolerance shows that the model performance isn't high. Training a new model on this data would just enhance the same biases as it is likely that any model trying to predict which outcomes are realistic would suffer from the same biases.

Instead you should try to adapt more of an "Online Learning" approach and Human supervision in the following steps:

1. Deploy your model and generate new predictions for actual, new data
2. Implement a Human supervision process that checks and proves the result of the prediction (large scale)
3. Use the results of this supervision to continually improve your model
4. Repeat 1-3 continually (Online Learning)
5. Use metadata from the Human supervision to implement a meta classification model marking likely outliers that need to be checked
6. Implement a small scale Human supervision process checking only the predictions marked by the meta model
7. Repeat

From a method perspective it would then be generally fine to build a classification model as you suggested. I would just use different data and a process.