How are 1x1 convolutions the same as a fully connected layer?

Question

I recently read Yan LeCuns comment on 1x1 convolutions:

In Convolutional Nets, there is no such thing as "fully-connected layers". There are only convolution layers with 1x1 convolution kernels and a full connection table.

It's a too-rarely-understood fact that ConvNets don't need to have a fixed-size input.

You can train them on inputs that happen to produce a single output vector (with no spatial extent), and then apply them to larger images.

Instead of a single output vector, you then get a spatial map of output vectors. Each vector sees input windows at different locations on the input. In that scenario, the "fully connected layers" really act as 1x1 convolutions.

I would like to see a simple example for this.

Example

Assume you have a fully connected network. It has only an input layer and an output layer. The input layer has 3 nodes, the output layer has 2 nodes. This network has $3 \cdot 2 = 6$ parameters. To make it even more concrete, lets say you have a ReLU activation function in the output layer and the weight matrix

$$ \begin{align} W &= \begin{pmatrix} 0 & 1 & 1\\ 2 & 3 & 5\\ \end{pmatrix} \in \mathbb{R}^{2 \times 3}\\ b &= \begin{pmatrix}8\\ 13\end{pmatrix} \in \mathbb{R}^2 \end{align} $$

So the network is $f(x) = ReLU(W \cdot x + b)$ with $x \in \mathbb{R}^3$.

How would the convolutional layer have to look like to be the same? What does LeCun mean with "full connection table"?

I guess to get an equivalent CNN it would have to have exactly the same number of parameters. The MLP from above has $2 \cdot 3 + 2 = 8$ parameters.

Answer 1

Your Example

In your example we have 3 input and 2 output units. To apply convolutions, think of those units having shape: [1,1,3] and [1,1,2], respectively. In CNN terms, we have 3 input and 2 output feature maps, each having spatial dimensions 1 x 1.

Applying an n x n convolution to a layer with k feature maps, requires you to have a kernel of shape [n,n,k]. Hence the kernel of your 1x1 convolutions have shape [1, 1, 3]. You need 2 of those kernels (or filters) to produce the 2 output feature maps. Please Note: $1 \times 1$ convolutions really are $1 \times 1 \times \text{number of channels of the input}$ convolutions. The last one is only rarely mentioned.

Indeed if you choose as kernels and bias:
$$ \begin{align} w_1 &= \begin{pmatrix} 0 & 1 & 1\\ \end{pmatrix} \in \mathbb{R}^{3}\\ w_2 &= \begin{pmatrix} 2 & 3 & 5\\ \end{pmatrix} \in \mathbb{R}^{3}\\ b &= \begin{pmatrix}8\\ 13\end{pmatrix} \in \mathbb{R}^2 \end{align} $$

The conv-layer will then compute $f(x) = ReLU\left(\begin{pmatrix}w_1 \cdot x\\ w_2 \cdot x\end{pmatrix} + \begin{pmatrix}b_1\\ b_2\end{pmatrix}\right)$ with $x \in \mathbb{R}^3$.

Transformation in real Code

For a real-life example, also have a look at my vgg-fcn implementation. The Code provided in this file takes the VGG weights, but transforms every fully-connected layer into a convolutional layers. The resulting network yields the same output as vgg when applied to input image of shape [244,244,3]. (When applying both networks without padding).

The transformed convolutional layers are introduced in the function _fc_layer (line 145). They have kernel size 7x7 for FC6 (which is maximal, as pool5 of VGG outputs a feature map of shape [7,7, 512]. Layer FC7 and FC8 are implemented as 1x1 convolution.

"Full Connection Table"

He might refer to a filter/kernel which has the same dimension as the input feature map. In both cases (Code and your Example) the spatial dimensions are maximal in the sense, that the spatial dimension of the filter is the same as the spatial dimension as the input.

Answer 2

A fully connected layer (for input size $n*n$ over with $i$ channels, and $m$ output neurons) IS NOT equivalent to a 1x1 convolution layer but rather to an $n$x$n$ convolution layer (i.e. a big kernel, same size as input- no pad) with number of filters equal to the FC output/hidden layer (i.e. $m$ filters)

As you asked, it has the same number of parameters as the FCN, i.e. $n*n*i*m$ (plus bias):

FCN: $n*n*i$ (weights per input layer= input$*$channels) $* m$ (times output/hidden layer width)

CNN: $n*n$ (each kernel) $* i $ (kernel per input channel) $* m$ (number of filters)

(Source)

Answer 3

The equivalent kernel simply has whatever shape the input has, and computes a tensor dot product. (I use the word "shape" as there seems to be some confusion over "size", which often ignores the channel/depth dimension). There's no "sliding the kernel across input" involved, since the kernel is as large as it can be. Quoting Stanford CS 231n course notes:

any FC layer can be converted to a CONV layer. For example, an FC layer with K=4096 that is looking at some input volume of size 7×7×512 can be equivalently expressed as a CONV layer with F=7,P=0,S=1,K=4096,F=7,P=0,S=1,K=4096. In other words, we are setting the filter size to be exactly the size of the input volume, and hence the output will simply be 1×1×4096 since only a single depth column “fits” across the input volume, giving identical result as the initial FC layer.

I believe "F=7,P=0,S=1,K=4096,F=7,P=0,S=1,K=4096" here means each conv kernel has shape 7x7x512, and there's 4096 such filters.

The earlier answer mentioned that the last fc of AlexNet (which receives input with shape 1x1x4096 and computes 1000 class scores) is implemented as "1x1 convolution". To be complete, each such conv kernel has shape 1x1x4096, and there's 1000 of them.

Le Cunn also explains this in the CNN paper, page 8, description of LeNet5:

Layer C5 is a convolutional layer with 120 feature maps. Each unit is connected to a 5x5 neighborhood on all 16 of S4's feature maps. Here because the size of S4 is also 5x5, the size of C5's feature maps is 1x1; this amounts to a full connection between S4 and C5.

Answer 4

To any newbie PyTorch user like me - do not confuse "fully connected layer" with a "linear layer".

A Linear layer and 1x1 convolutions are the same thing. It took me awhile to understand that there is no such thing as a "fully connected layer" - it's simply a flattening of spatial dimensions into a 1D giant tensor. The flattening causes every element on all spatial locations to be presented as a separate feature map/channel/dimension (whatever you want to call it). Hence, when we apply a linear layer on a flattened tensor, we get a fully connected operation, which obviously, is not the same as a 1x1 convolution.

A 1x1 convolution is simply a weighted summation of input feature maps, in OP's question, 3. O[0] = $\sum_{j=0}^{1} X[j]W_0[j] + b[0]$. Similarly, O[1] = $\sum_{j=0}^{1} X[j]W_1[j] + b[1]$ and O[2] = $\sum_{j=0}^{1} X[j]W_2[j] + b[2]$.

The above 3 equations open up to become $O = W_{2 \times 3}X_{3 \times 1} + b_{2 \times 1}$.

Other answers which claim that a fully-connected layer is simply a valid nxn convolution layer for an input of size nxn is confusing (albeit, correct).