Important CNN architectures

Question

What are some of the important CNN architectures one should know about? What CNN architectures did well on the ImageNet ILSVRC challenge? What CNN architectures are good candidates for transfer learning?

Answer 1

Here is a list of several important CNN architectures:

• LeNet-5 the first successful application of CNNs to digit recognition, developed by Yann LeCun in 1990. It consists of a sequence of Convolutional, ReLU, Max Pooling layers followed by a Fully Connected layer. Reference: Y. LeCun et. al., "Gradient-Based Learning Applied to Document Recognition", Proceedings of the IEEE, 1998
• AlexNet popularized CNNs in computer vision, did really well on the ImageNet ILSVRC challenge in 2012 showing significant gains in performance. The network has similar architecture to LeNet but is deeper and bigger and features convolutional layers stacked on top of each other. Reference: A. Krizhevsky, et. al. "ImageNet Classification with Deep Convolutional Neural Networks", NIPS, 2012
• VGG16 demonstrated the importance of depth as a critical component to good performance, it was a runner-up in ILSVRC 2014. The architecture consists of a stacked convolutional and max pooling layers with increasing depth and it uses a large number of parameters due to the final fully connected layers. Reference: K. Simonyan, "Very Deep Convolutional Networks for Large-Scale Image Recognition", ICLR, 2015
• Inception original version called GoogLeNet from Google won the ILSVRC 2014 challenge the architecture consists of inception modules that dramatically reduce the number of parameters, it uses multi-scale 3x3, 5x5 convolutional filters including 1x1 convolutions for dimensionality reduction. Reference: C. Szegedy et al., "Going Deeper with Convolutions", CVPR, 2015
• ResNet50 residual network was the winner of ILSVRC 2015, it introduces skip connections for easier training that enable very deep architectures and makes use of batch normalization. Reference: K. He et al., "Deep Residual Learning for Image Recognition", CoRR, 2015

Answer 2

See my masters thesis. Chapter 2.2 and 2.3 for a high-level overview over the building blocks. Appendix D describes a couple of really well known CNN architectures in detail:

• Lenet-5 (historic)
• AlexNet (still in use, but not SotA)
• VGG-16 D (Still in use, but not SotA)
• Googlenet and the Inception-Nets (v2, v3, v4)

Not described in detail, but still important

• Resnets
• Densenets