I am interested in any data, publications, etc about what is the smallest neural network that can achieve a certain level of classification performance. By small I mean few parameters, not few arithmetic operations (=fast).
I am interested primarily in convolutional neural networks for vision applications, using something simple like CIFAR-10 without augmentation as the benchmark. Top-performing networks on CIFAR in recent years have had anywhere between 100 million and 0.7 million parameters (!!), so clearly small size is not (always) a bad thing.
Small networks are also in general faster to train and overfit less. Moreover, recent work on Knowledge Distillation, FitNets, etc show ways of making smaller networks from large networks while preserving most of the performance.
Another question is, what is the best performance achievable with a network no larger than a fixed size?
Examples of especially small networks that get good performance (100k parameters with 10% on CIFAR, anyone?) or systematic studies of the size vs performance tradeoff would be appreciated.