I am reading this paper - https://arxiv.org/abs/1503.02531v1 - devoted to knowledge distillation in neural networks.
One interesting approach is mentioned in this paper in sections
5.4 - to create smaller specialists models, which are trained to distinguish only a subset of labels and mark all other labels as some dummy/background class.
The most straightforward application would be to split the data in some large dataset, for instance
Imagenet, with large number of classes into smaller parts with images containing similar in some classes, grouped by a certain meaningful criterion. Then one creates a separate model for this subset of labels, for example, model which is trained to distinguish cars, or model which identifies different cat species.
On the inference stage when passing the image one gets probabilities for presence of certain classes, and chooses n-most confident models and aggregates in a certain ways outputs of these models. Another option would be to create a general purpose model -
generalist - and pass the input image to the specialists, according to the probabilities , obtained by the generalist.
The advantage of this approach is that each model would learn features pertinent to the concrete task. At the same time, multi-purpose model learns about a lot of classes, but knowledge of features, relevant to different classes may interfere with each other.
Disadvantages and issues
Due to seeing less data, than the general purpose model, each of the specialist model is more prone to overfitting, as mentioned in the paper. However, superficially solving less sophisticated tasks models can be created much simplier, shallower and containing less layers. Ensembling effect from different specialists in case of partially overlapping domains- seems to remedy at a certain extent overfitting.
Another problem is an increase possible increase of the inference time, because one needs to evaluate several models for each sample. On the other hand, each of the model is smaller, and in prior it is difficult to say, whether evaluation of several simpler models would be more computationally expensive than the one pass through big model.
Moreover, large models, trained on
Imagenet are useful as a feature extractors for various problems in
Computer Vision tasks and in
Transfer learning, whereas this ensemble of experts would not be useful in this case.
Nevertheless, this specialists ensemble approach seems to be interesting, but I've not seen a lot of literature devoted to this topic. Did this approach turn out to be unsuccessful or inefficient?