I am reading this paper - https://arxiv.org/abs/1503.02531v1 - devoted to knowledge distillation in neural networks.

One interesting approach is mentioned in this paper in sections 5.3, 5.4 - to create smaller specialists models, which are trained to distinguish only a subset of labels and mark all other labels as some dummy/background class.

The most straightforward application would be to split the data in some large dataset, for instance Imagenet, with large number of classes into smaller parts with images containing similar in some classes, grouped by a certain meaningful criterion. Then one creates a separate model for this subset of labels, for example, model which is trained to distinguish cars, or model which identifies different cat species.

On the inference stage when passing the image one gets probabilities for presence of certain classes, and chooses n-most confident models and aggregates in a certain ways outputs of these models. Another option would be to create a general purpose model - generalist - and pass the input image to the specialists, according to the probabilities , obtained by the generalist.


The advantage of this approach is that each model would learn features pertinent to the concrete task. At the same time, multi-purpose model learns about a lot of classes, but knowledge of features, relevant to different classes may interfere with each other.

Disadvantages and issues

Due to seeing less data, than the general purpose model, each of the specialist model is more prone to overfitting, as mentioned in the paper. However, superficially solving less sophisticated tasks models can be created much simplier, shallower and containing less layers. Ensembling effect from different specialists in case of partially overlapping domains- seems to remedy at a certain extent overfitting.

Another problem is an increase possible increase of the inference time, because one needs to evaluate several models for each sample. On the other hand, each of the model is smaller, and in prior it is difficult to say, whether evaluation of several simpler models would be more computationally expensive than the one pass through big model.

Moreover, large models, trained on Imagenet are useful as a feature extractors for various problems in Computer Vision tasks and in Transfer learning, whereas this ensemble of experts would not be useful in this case.

Nevertheless, this specialists ensemble approach seems to be interesting, but I've not seen a lot of literature devoted to this topic. Did this approach turn out to be unsuccessful or inefficient?

  • $\begingroup$ A simple way to roughly estimate how successful an approach is is to look at the citations: According to Google Scholar this paper has more than 5k citations, so there is actually quite a lot of literature around this approach. $\endgroup$ – Erwan Jan 13 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.