What I would like to do is recursively:
- Train the model on all data
- Remove the sample(s) with highest error
- Repeat until the remaining samples have an acceptable error
The hypothesis is: "To maximize production performance, we should filter the training set by removing samples with high cross-validation error, assuming they have bad data."
The downside is you'll never know whether you removed something that is valid and hard to predict, or something with messed up data - but my hope is this will overall stabilize model performance in production.
I can find zero information on if this is a valid technique. What I would like to know is what the best way to evaluate my training data filtering, and is this a completely insane thing to do?
More details:
I do machine learning in the construction industry, which has significant variability in both data quality and the types of tasks which are being predicted. I am training a CatBoost model on hierarchical data that looks like this:
All Piping Tasks
- 6" pipe
- steel
- copper
- 8" pipe
- copper
- pvc
- etc.
All piping tasks are placed in the same model, as I often have only a few samples of each specific piping type. My challenge is, it may turn out an 8" copper pipe sample has bad data, which pollutes the predictions of the other piping types. There is no way to confirm whether the data is bad or good, as it was manually entered into the source system. Outlier detection helps somewhat but there is generally a large amount of variability naturally baked into the data.