What is actually the best neural network architecture for the classic
MNIST digit classifying task? I couldn't find any that would claim to be the winner...
Actually, what I'm going to discuss is not an architecture and is like a module used in networks. It performs really well on character based data-sets like
MNIST although there is other functionality for them. There is a paper called Spatial Transformer Networks written by Max Jaderberg et al. It tries to introduce an alternative for pooling layers. What it does is tries to find the canonical shape of its input by reducing transformations, like translation and rotation, or even diminishing the distortion of the inputs. It introduces a module which helps convolutional networks to really be spatial invariant. The work is amazing and the paper is easy to be read. Based on experience, using these modules may reduce the number of neurons and layers, because networks employing them won't try to learn extra stuff of the inputs, like applied transformations to the inputs. The reason is that the net will try to learn the canonical shape of inputs.
(a) shows the arbitrary input to the network (b) shows what spatial transformer has done and finally (c) is the output of the spatial transformer which can be used through other layers of the networks.
One of the significant achievements of this module is that it tries to enhance distorted inputs.
Its performance can be seen here.
As far as I know, the state-of-the-art architecture in MNIST competitions is the 2018 RMDL (Random Multimodel Deep Learning for classification) model. Check the ArXive paper.
You need to take a look at the state of the art (SOTA) visualisations on Papers with Code.
An ensemble of 15 CNNs classify Kaggle's MNIST digits after training on Kaggle's 42,000 images plus 25 million more images created by rotating, scaling, and shifting Kaggle's images. Learning from 25,042,000 images, this ensemble of CNNs achieves 99.75% classification accuracy. Techniques include data augmentation, nonlinear convolution layers, learnable pooling layers, ReLU activation, ensembling, bagging, decaying learning rates, dropout, batch normalization, and adam optimization. GM CDeotte explains the arch in great details here: https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist
The SOTA link shared in the other answer uses homogeneous vector capsules generated from the final filters and gives 99.85% - a full 0.1% increase. However it is not a plain vanilla Apple to Apple conversion as the Kaggle solution involves using only a subset of the MNist images provided by the hosts. Possibly using it on the full MINST database could take it closer to the SOTA score.
The real question though is - how does this compare with the accuracy of the best SOTA CNN architecture in the universe - the human mind. While there is no formal study of the same, 99.8% was the 'claimed' human benchmark and it seems to have been broken now