I am building a fair amount of statistical models - text classifiers and sequence taggers. The statistical models are linear in features - Logistic Regression and linear chain first order CRF.
The models undergo several stages starting from a rough, initial version all the way to a mature production model. Given a specification and some dataset (often with incomplete annotation, noisy labels and features), I build the first model from a small, hand labeled dataset. This dataset is then grown iteratively by adding more labeled examples (e.g. using crowdsourcing).
Then comes the test phase. The statistical model is actively evaluated within a larger system by testers. I, as a developer, start getting the failure reports on their input. I am supposed to fix the model, update it in the system and deploy it.
Every individual error report is fairly small to manually go through and fix every misclassification. Obviously it is incorrect just to simply add the failure cases with correct labels to the training set. You never know if the model learned that new pattern. I do this by comparing the failure cases with a vast unlabeled dataset of examples (real usage data but no labels) and extract similar examples. I label those similar examples, add to the training set and re-train the model. Then I check if the new model fixes the originally reported issues.
My question is: Is this a correct approach to build and continuously improve the statistical models? May be there are some hidden caveats that will bias my model in some sense?