Selecting Training Set from Model CV Error

Question

What I would like to do is recursively:

1. Train the model on all data
2. Remove the sample(s) with highest error
3. Repeat until the remaining samples have an acceptable error

The hypothesis is: "To maximize production performance, we should filter the training set by removing samples with high cross-validation error, assuming they have bad data."

The downside is you'll never know whether you removed something that is valid and hard to predict, or something with messed up data - but my hope is this will overall stabilize model performance in production.

I can find zero information on if this is a valid technique. What I would like to know is what the best way to evaluate my training data filtering, and is this a completely insane thing to do?

More details:
I do machine learning in the construction industry, which has significant variability in both data quality and the types of tasks which are being predicted. I am training a CatBoost model on hierarchical data that looks like this:

All Piping Tasks

• 6" pipe
  • steel
  • copper
• 8" pipe
  • copper
  • pvc
• etc.

All piping tasks are placed in the same model, as I often have only a few samples of each specific piping type. My challenge is, it may turn out an 8" copper pipe sample has bad data, which pollutes the predictions of the other piping types. There is no way to confirm whether the data is bad or good, as it was manually entered into the source system. Outlier detection helps somewhat but there is generally a large amount of variability naturally baked into the data.

Answer 1

This is not insane at all, but as you correctly identified there's a risk to remove valid instances.

If you have some annotated data this risk can be estimated simply by counting the proportion of valid instances which have been eliminated by this method. But at the end of the day the main indicator is the overall performance obtained by the final model.

I can find zero information on if this is a valid technique.

I think you would find information about this kind of method in the area of semi-supervised learning. The standard case is slightly different but there are some similarities: the iterative training and the unknown status of the instances.

Answer 2

Further research indicates the most relevant technique to address this problem is data valuation, also known as data shapley. Here are 2 relevant papers on the topic:
Data Shapley: Equitable Valuation of Data for Machine Learning
What is your data worth? Equitable Valuation of Data

I spent a week implementing this technique on my project. It is highly effective at ranking training data from least useful to most useful in terms of performance on the test set. I then review the lowest value records for data quality problems, of which there are frequently obvious ones. If you have an acceptable test set to optimize against this is a great technique.

The downsides are:

1. You need a clean test set that is representative of the population. If you're using cross validation, you may be evaluating against bad cv data points.
2. Most examples in the paper are classification, while my problem is regression. It is much easier to identify a mis-labeled image than a continuous target.
3. Extremely expensive compute. You'll end up training anywhere from 3N to 10N models, where N is the number of samples in the training dataset.
4. No library exists to do it. I ended up grabbing chunks of code from the paper's github, but you'll likely need to implement something yourself.

Here is most striking result I got:

Each line is a different model, at 0 all of the training data is included. 10 means we have removed 10% of the training data, starting with the lowest value records. As we remove more low value data, the test RMSE decreases. All RMSE's have been normalized so all models can be plotted together.