Getting Validation Accuracy of 99% with MNIST with less than 10000 parameters CNN

Question

Given MNIST dataset in keras,the challenge is to develop a CNN neural net model with less than 10k parameters with 99% validation accuracy.

I tried making the model for the same but am getting accuracy as 98.71.
Please find below the code for the same.

    def create_model():
        lay1=Conv2D(2,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay1=Conv2D(2,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay1)
        lay1=Conv2D(2,kernel_size=(1,1),activation='relu',padding='same')(lay1)
        lay1=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay1)
        lay2=Conv2D(4,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay2=Conv2D(4,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay2)
        lay2=Conv2D(4,kernel_size=(1,1),activation='relu',padding='same')(lay2)
        lay2=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay2)
        lay3=Conv2D(6,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay3=Conv2D(6,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay3)
        lay3=Conv2D(6,kernel_size=(1,1),activation='relu',padding='same')(lay3)
        lay3=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay3)
        fc=concatenate([lay1,lay2,lay3])
        fc=Flatten()(fc)
        fc=Dense(10,activation='relu')(fc)
        outputs=Dense(10,activation='softmax')(fc)
        model=Model(input=inputs,output=outputs)
        model.compile(loss=keras.losses.categorical_crossentropy,
          optimizer=keras.optimizers.Adadelta(),
          metrics=['accuracy'])
        return model

The total parameters coming are 8,862 and the batch size used for the above is 32 and the number of epochs are 10.

Can you please suggest ways to improve the model with the constraints on the number of parameters so that the validation accuracy is 99% or above?

Answer 1

Policy distillation is an excellent way to efficiently reduce number of parameters and maintain the quality of the original (large) network.

According to the paper, while having orders of magnitude less numbers, distilled models can actually improve score from 84% up to 124% and above.

The basic idea is that we train a new, smaller network from scratch, to predict what the original (large) network would produce.

An analogy would be: it's a lot easier to assign the desired mindset to a young child, than to update an old person's way of thinking.

Answer 2

You could use some batch normalisation layer (before or after each activation function, there is a debat) and use dropout.

Answer 3

Please try to use dropout function and image Augmentation simply amends your images on-the-fly while training using transforms like rotation. So, it could 'simulate' an image by rotating a 'standing' cat by 90 degrees. As such you get a cheap way of extending your dataset beyond what you have already.

Image augmentation and dropout - Key techniques to fight overfitting in computer vision tasks to incorporate into the data pipeline and image classifier model.