I think the confusion with the Inception module is the somewhat complicated structure. The point on the relevant CS231n slide (#37), saying there are no FC layers is partially correct. (Remember this is only a summary of the model to get the main points across!). In the actual part of the model being explained on that slide, they are referring only to the Inception modules:
No FC layers!
Definitions will, however, play a big role in deciding whether or not there are FC layers in the model.
In the bigger scheme of things (beyond a single Inception module), we have first to distinguish between the train and test time architectures.
At train time there are auxilliary branches, which do indeed have a few fully connected layers. These are used to force intermediate layers (or inception modules) to be more aggressive in their quest for a final answer, or in the words of the authors, to be more discriminate.
From the paper (page 6 [Szegedy et al., 2014]):
One interesting insight is that the strong performance
of relatively shallower networks on this task suggests that the features produced by the layers in the
middle of the network should be very discriminative. By adding auxiliary classifiers connected to
these intermediate layers, we would expect to encourage discrimination in the lower stages in the
classifier, increase the gradient signal that gets propagated back, and provide additional regularization.
The slice of the model shown below displays one of the auxilliary classifiers (branches) on the right of the inception module:
This branch clearly has a few FC layers, the first of which is likely followed by a non-linearity such as a ReLU or tanh. The second one simply squishes the 1000 input weights into whatever number of classes are to be predicted (coincidentally or not, this is a 1000 here for ImageNet).
However, at test time, these branches are not active. They were used simply to train the weights of the modules, but do not contribute to the final classification probabilities produces at the end of the entire model architecture.
This all leaves us with just the suspsicious looking block right at the end of the model:
There is clearly a big blue FC layer there!
This is where definitions come into play. It is somewhat subjective.
Is a fully connected layer one in which each $m$ weight is connected to each of $n$ nodes? Is it a layer in which representation are learned, and if so, does the layer require a non-linearity? We know that neural networks requires the non-linearities, such as ReLU and tanh functions to be applied to the outputs of a layer (thinking in forward flow). Without these, neural networks would simply be a combinations of linear functions, and so going deeper wouldn't theoretically add any power as we essentially would just be performing a huge linear regression.
In this spirit, we can look at the final piece of the puzzle, and point out that tis final FC layer is noted to simply be linear! That is, it takes all the weights resulting from the preceding Average Pooling layer, and combines them into a linear combination of only 1000 values - ready for the softmax. This can all be understood from the tabular overview of the network architecture:
So, do you agree with the stanford guys or not? I do!
