1
$\begingroup$

I am building a fair amount of statistical models - text classifiers and sequence taggers. The statistical models are linear in features - Logistic Regression and linear chain first order CRF.

The models undergo several stages starting from a rough, initial version all the way to a mature production model. Given a specification and some dataset (often with incomplete annotation, noisy labels and features), I build the first model from a small, hand labeled dataset. This dataset is then grown iteratively by adding more labeled examples (e.g. using crowdsourcing).

Then comes the test phase. The statistical model is actively evaluated within a larger system by testers. I, as a developer, start getting the failure reports on their input. I am supposed to fix the model, update it in the system and deploy it.

Every individual error report is fairly small to manually go through and fix every misclassification. Obviously it is incorrect just to simply add the failure cases with correct labels to the training set. You never know if the model learned that new pattern. I do this by comparing the failure cases with a vast unlabeled dataset of examples (real usage data but no labels) and extract similar examples. I label those similar examples, add to the training set and re-train the model. Then I check if the new model fixes the originally reported issues.

My question is: Is this a correct approach to build and continuously improve the statistical models? May be there are some hidden caveats that will bias my model in some sense?

$\endgroup$
1
$\begingroup$

No offense, but let me guess, you work for a large company where there is no person with a full understanding of the process overseeing this? I say this because it sounds as if the testers want you to manually and intentionally over-fit your models. There is no such thing as a perfectly accurate model.

Yes, I would say that there is bias in your model building. New model input should be a random selection not by finding and labeling similar instances to ones that were misclassified.

Your production model will degrade over time (the distribution in the population has changed). You should have a sense of a threshold, an error rate at which is no longer exceptable and you will need to retrain. To retrain, you could add a random selection of new labeled data to your existing data, then train, validate, select model, deploy, and monitor.

You may "fix" an error report with a new retraining, but it is an unrealistic expectation that this will also cover all previous error reports. Your model needs to generalize. See here:

https://en.wikipedia.org/wiki/Test_set

Look at the image at the bottom. In essence you are being asked to create the orange model were you want to create the green model, which generalizes better to unseen data.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the feedback, this is basically the line of thoughts I had. In practice the number of added examples is very small compared to the complete dataset size. I would need to add an AWFUL amount of such small fixes to bias it, I believe. My question to you is how the model's short comings can be fixed then? I cannot just throw some random data (cannot afford to label large number of arbitrary examples) and hope it works for that one fix. Not speaking about other small issues. $\endgroup$ – Vladislavs Dovgalecs Nov 3 '15 at 16:33
  • $\begingroup$ That is part of the point, it might not get fixed, which should be an expectation. If the misclassification rate is within your pre-defined threshold, then it is an expected error. If the rate is above the threshold you may need to retrain. My advice is expect errors, set a threshold, retrain. It is sometimes possible to automate labeling. I don't know what models you are using, but from the sounds of it a well defined rules engine like Drools sounds like a viable alternative. Your models could help drive the rules. $\endgroup$ – user13684 Nov 3 '15 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.