# Large general-purpose model vs ensemble of many smaller models

I am reading this paper - https://arxiv.org/abs/1503.02531v1 - devoted to knowledge distillation in neural networks.

One interesting approach is mentioned in this paper in sections 5.3, 5.4 - to create smaller specialists models, which are trained to distinguish only a subset of labels and mark all other labels as some dummy/background class.

The most straightforward application would be to split the data in some large dataset, for instance Imagenet, with large number of classes into smaller parts with images containing similar in some classes, grouped by a certain meaningful criterion. Then one creates a separate model for this subset of labels, for example, model which is trained to distinguish cars, or model which identifies different cat species.

On the inference stage when passing the image one gets probabilities for presence of certain classes, and chooses n-most confident models and aggregates in a certain ways outputs of these models. Another option would be to create a general purpose model - generalist - and pass the input image to the specialists, according to the probabilities , obtained by the generalist.

The advantage of this approach is that each model would learn features pertinent to the concrete task. At the same time, multi-purpose model learns about a lot of classes, but knowledge of features, relevant to different classes may interfere with each other.

Moreover, large models, trained on Imagenet are useful as a feature extractors for various problems in Computer Vision tasks and in Transfer learning, whereas this ensemble of experts would not be useful in this case.