4
$\begingroup$

Let's say that I have a small or medium sized dataset of images, say 50,000. I use transfer learning to train a deep learning classification model. Call this model A. Model A is deemed to have good enough performance to be deployed. I deploy model A to a production environment where many users are able to consume the service by sending an image to an endpoint and receiving back the predicted class.

Now lets say the service becomes very popular, and after a few months, model A has labeled a few million images. Assume that these images and their labels as assigned by A are maintained in a new dataset. Say I train a new classification model. Call this model B. This model uses the new dataset consisting of the millions of images labeled by model A.

Model B has trained on much, much more data than did model A; however, the training data for model B was labeled exclusively by model A, so it will have label noise in accordance with model A's errors.

Is there any reason to believe that model B should show an improvement in performance over A, or will B merely learn the errors of A and fail to show any improvement?

If B won't/shouldn't improve over A in the given scenario, are there any slight tweaks to the scenario that might allow for B to show a performance gain over A?

================================================================

I have to believe that this problem is extremely common in industry, but I have not found a canonical or even common approach to solving it. For example, a similar question exists here for numerical data, but it didn't get much traction.

I'm interested in any attempted solution or a method that is commonly used in industry for image classifiers. Feel free to rephrase the question in your proposed solution. The only necessary assumptions you must make when drafting a solution are that 1) you've trained some sort of initial model(s) on an initial image dataset, 2) you now you have a huge number of new images that can be considered either unlabeled or labeled by the initial model(s), and 3) you can only consider automated solutions, there can be no manual checking of the data.

$\endgroup$

1 Answer 1

6
$\begingroup$

For strictly supervised methods, I would not expect model B to perform significantly better than model A. The only way model B can do better than model A is by correctly classifying samples for which model A produces an incorrect classification. But since model B is trained on model A predictions, this is exactly what model B tries to avoid - it should attempt to produce the same incorrect classifications as model A, which are "correct" as far as model B is concerned. To improve on model A, model B must "intend" to be wrong.

For model B to perform better, it must selectively "misclassify" samples that model A got wrong. But neither model knows what samples those are, if they did, they would be able to recognize the error and not have gotten them wrong in the first place.

Model B could potentially do better with semi-supervised methods that use both a set of provided labels as well as the structure of the data in feature space. Such methods may allow some flexibility in the training labels, allowing the optimization to consider things other than how many training labels are correctly predicted. With supervised methods, model B can do no better than exactly recapitulating model A's output, but with semi-supervised methods, model B may be optimized in a way that produces labels distinct from model A.

$\endgroup$
2
  • $\begingroup$ Seems reasonable. Semi-supervised learning is something I'll look into. What if the scenario changes and instead of having a single initial model A, we have several initial models A1 ... A10. Each image is randomly classified by one of the 10 initial models, then their outputs are pooled to train B. B can't replicate all of their errors since they will have slightly different errors. Now can B improve over what a single initial A would be? $\endgroup$
    – AJV
    May 16 at 15:09
  • 1
    $\begingroup$ @AJV Yes, this is the idea behind ensemble classifiers - a simple example is majority voting among multiple classifiers. If you have 3 independent classifiers each with 60% accuracy, for example, a majority vote among the three will have 65% accuracy. "Boosting" is is one such method of training so-called weak classifiers that can be combined to have performance that's potentially much better than any one alone. $\endgroup$ May 16 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.