# Number of Fully connected layers in standard CNNs

I have a question targeting some basics of CNN. I came across various CNN networks like AlexNet, GoogLeNet and LeNet.

I read at a lot of places that AlexNet has 3 Fully Connected layers with 4096, 4096, 1000 layers each. The layer containing 1000 nodes is the classification layer and each neuron represents the each class.

Now I came across GoogLeNet. I read about its architecture at http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture9.pdf It says that GoogLeNet has 0 FC layers. However, you do need the 1000 node layer in the end with Softmax activation for the classification task. So the final layer isn't treated as FC in this?

Also then, what is the number of FC layers in LeNet-5?

A bit confused. Any help or leads would be greatly appreciated.

I think the confusion with the Inception module is the somewhat complicated structure. The point on the relevant CS231n slide (#37), saying there are no FC layers is partially correct. (Remember this is only a summary of the model to get the main points across!). In the actual part of the model being explained on that slide, they are referring only to the Inception modules:

No FC layers!

Definitions will, however, play a big role in deciding whether or not there are FC layers in the model.

In the bigger scheme of things (beyond a single Inception module), we have first to distinguish between the train and test time architectures.

At train time there are auxilliary branches, which do indeed have a few fully connected layers. These are used to force intermediate layers (or inception modules) to be more aggressive in their quest for a final answer, or in the words of the authors, to be more discriminate.

From the paper (page 6 [Szegedy et al., 2014]):

One interesting insight is that the strong performance of relatively shallower networks on this task suggests that the features produced by the layers in the middle of the network should be very discriminative. By adding auxiliary classifiers connected to these intermediate layers, we would expect to encourage discrimination in the lower stages in the classifier, increase the gradient signal that gets propagated back, and provide additional regularization.

The slice of the model shown below displays one of the auxilliary classifiers (branches) on the right of the inception module:

This branch clearly has a few FC layers, the first of which is likely followed by a non-linearity such as a ReLU or tanh. The second one simply squishes the 1000 input weights into whatever number of classes are to be predicted (coincidentally or not, this is a 1000 here for ImageNet).

However, at test time, these branches are not active. They were used simply to train the weights of the modules, but do not contribute to the final classification probabilities produces at the end of the entire model architecture.

This all leaves us with just the suspsicious looking block right at the end of the model:

There is clearly a big blue FC layer there!

This is where definitions come into play. It is somewhat subjective. Is a fully connected layer one in which each $m$ weight is connected to each of $n$ nodes? Is it a layer in which representation are learned, and if so, does the layer require a non-linearity? We know that neural networks requires the non-linearities, such as ReLU and tanh functions to be applied to the outputs of a layer (thinking in forward flow). Without these, neural networks would simply be a combinations of linear functions, and so going deeper wouldn't theoretically add any power as we essentially would just be performing a huge linear regression.

In this spirit, we can look at the final piece of the puzzle, and point out that tis final FC layer is noted to simply be linear! That is, it takes all the weights resulting from the preceding Average Pooling layer, and combines them into a linear combination of only 1000 values - ready for the softmax. This can all be understood from the tabular overview of the network architecture:

So, do you agree with the stanford guys or not? I do!

There are 2 fully connected layers in LeNet-5. It is shown in Figure 2 of "Gradient-Based Learning Applied to Document Recognition":

• That helped. Thank you. What about googlenet? It has FC layer? – Abhishek Saxena May 3 '18 at 20:09

I just checked the original inception network paper (GoogLeNet) here and it does have a fully connected layer before the softmax.

• Yes it does have. But then why is it mentioned 0 FC over here: cs231n.stanford.edu/slides/2017/cs231n_2017_lecture9.pdf – Abhishek Saxena May 3 '18 at 23:30
• Like the other answer mentioned, I think they are talking about the bottleneck architecture rather than the entire network. It is absurd to say that a fully functional classification model does not have a fully connected layer. – Sanjay Krishna May 5 '18 at 15:56