Is this a correct way improving a statistical model?

Question

I am building a fair amount of statistical models - text classifiers and sequence taggers. The statistical models are linear in features - Logistic Regression and linear chain first order CRF.

The models undergo several stages starting from a rough, initial version all the way to a mature production model. Given a specification and some dataset (often with incomplete annotation, noisy labels and features), I build the first model from a small, hand labeled dataset. This dataset is then grown iteratively by adding more labeled examples (e.g. using crowdsourcing).

Then comes the test phase. The statistical model is actively evaluated within a larger system by testers. I, as a developer, start getting the failure reports on their input. I am supposed to fix the model, update it in the system and deploy it.

Every individual error report is fairly small to manually go through and fix every misclassification. Obviously it is incorrect just to simply add the failure cases with correct labels to the training set. You never know if the model learned that new pattern. I do this by comparing the failure cases with a vast unlabeled dataset of examples (real usage data but no labels) and extract similar examples. I label those similar examples, add to the training set and re-train the model. Then I check if the new model fixes the originally reported issues.

My question is: Is this a correct approach to build and continuously improve the statistical models? May be there are some hidden caveats that will bias my model in some sense?

Answer 1

No offense, but let me guess, you work for a large company where there is no person with a full understanding of the process overseeing this? I say this because it sounds as if the testers want you to manually and intentionally over-fit your models. There is no such thing as a perfectly accurate model.

Yes, I would say that there is bias in your model building. New model input should be a random selection not by finding and labeling similar instances to ones that were misclassified.

Your production model will degrade over time (the distribution in the population has changed). You should have a sense of a threshold, an error rate at which is no longer exceptable and you will need to retrain. To retrain, you could add a random selection of new labeled data to your existing data, then train, validate, select model, deploy, and monitor.

You may "fix" an error report with a new retraining, but it is an unrealistic expectation that this will also cover all previous error reports. Your model needs to generalize. See here:

https://en.wikipedia.org/wiki/Test_set

Look at the image at the bottom. In essence you are being asked to create the orange model were you want to create the green model, which generalizes better to unseen data.