What is actually the best neural network architecture for the classic
MNIST digit classifying task? I couldn't find any that would claim to be the winner...
Actually, what I'm going to discuss is not an architecture and is like a module used in networks. It performs really well on character based data-sets like
MNIST although there is other functionality for them. There is a paper called Spatial Transformer Networks written by Max Jaderberg et al. It tries to introduce an alternative for pooling layers. What it does is tries to find the canonical shape of its input by reducing transformations, like translation and rotation, or even diminishing the distortion of the inputs. It introduces a module which helps convolutional networks to really be spatial invariant. The work is amazing and the paper is easy to be read. Based on experience, using these modules may reduce the number of neurons and layers, because networks employing them won't try to learn extra stuff of the inputs, like applied transformations to the inputs. The reason is that the net will try to learn the canonical shape of inputs.
(a) shows the arbitrary input to the network (b) shows what spatial transformer has done and finally (c) is the output of the spatial transformer which can be used through other layers of the networks.
One of the significant achievements of this module is that it tries to enhance distorted inputs.
Its performance can be seen here.
As far as I know, the state-of-the-art architecture in MNIST competitions is the 2018 RMDL (Random Multimodel Deep Learning for classification) model. Check the ArXive paper.
You need to take a look at the state of the art (SOTA) visualisations on Papers with Code.