Convolution layer dimensions in deeper layers?

Question

I am trying to understand the CNN network dimensions:

layer                   activation shape    number of weights 

Input                   (25, 64, 1)         0
Conv2D(7x7, 1, 100)     (25, 64, 100)       4.900 (7x7x100)
MaxPool2D               (13, 32, 100)       0
Conv2D(5x5, 1, 150)     (13, 32, 150)       375.000 (5x5x150x100) 
...

The input dimension is 25x64, then first Conv layer applies 100 convolutions - so the output dimension is 25x64x100. Max pooling reduces this to 13x32x100.

Second Conv layer applies 150 convolutions. What's unclear to me is the dimension after the second Conv layer. Shouldn't it be 13x32x150x100 (instead of just 13x32x150)? How is the convolution layer applied on a 3D input?

Number of weights for the first Conv layer is 7x7x100, and for the second layer it is 5x5x150x100, meaning that weights are saved for each of the 100 input layers from the previous conv. This also leads me to think that output dimension should be 4D.

Answer 1

The output shapes for convolutional layers are calculated in the following way:

Let's say that the input shape for some convolutional layer is WxHxC:

• W - width
• H - height
• C - channels

Now, assume that you have only one convolutional kernel, eg. size 5x5 (width and height). That kernel will actually be of size 5x5xC (C is the number of channels in the input shape) because one kernel must multiply all channels in the input when it is fixed in some place of the input. As you know, for that one fixed position, you get only one output number. When you repeat this for all input positions, you get only one output feature map (ie. with shape WxHx1 assuming that you keep the input dimensions by using padding ...).

In order to get more feature maps on the output, you need to repeat the process with new convolutional kernels (all those kernels must have the channel number equal to input shape channels). So, if you repeat this K times, your output feature map will have dimensions WcHxK. And the size of the kernel for that convolutional layer is said to be 5x5xCxK.

You can consult the following image:

In the image, one convolutional kernel (in the image it's called "Convnet Filter") outputs one output feature map (in the image it's called "One Feature Map").

Hence, in your case, for the first convolutional layer, the size of kernel is 7x7x1x100, and the output shape is 25x64x100. For the second convolutional layer, kernel size is 5x5x100x150 and the output shape is 13x32x150.

---

The above-described convolution is the "classic" convolution. There are other types of convolution, eg. Depthwise separable convolution, which use different ways of calculation. Consult A Basic Introduction to Separable Convolutions tutorial for more information about different types of convultion (it includes also descrption of "classic" convolution in the "Normal Convolution" section).

Answer 2

Second Conv layer applies 150 convolutions. What's unclear to me is the dimension after the second Conv layer. Shouldn't it be 13x32x100x150? How is the convolution layer applied on a 3D input?

One key idea behind CNNs is to restrict the number of connections compared to fully connected layers. This is, in part, done by using kernels which define a window size of spatial inputs to be considered:

As you can see a given green neuron does not consider all blue inputs but only a subset defined by the kernel.

However, what this illustration does not show is that the whole input depth is considered. Each neuron of the green layer in the image above is not only connected to $3 \cdot 3$ blue inputs but $3 \cdot 3 \cdot depth_{input}$ blue inputs.

That is why a conv. layer outputs as many feature maps (which is equivalent to the depth of the conv. layer) as it applies kernels. Accordingly your second conv. layer will have an output of depth 150.

Number of weights for the first Conv layer is 7x7x100, and for the second layer it is 5x5x150x100, meaning that weights are saved for each of the 100 input layers from the previous conv.

If you pick a feature map $F$ of a conv. layer: this feature map uses the same weights across the spatial input dimensions, i.e. it needs $K_{widht} \cdot K_{height}$ weights (where $K$ is the kernel size). This means the green neurons in the above image share their weights!

However, as explained earlier it not only takes the spatial input shown in that image but considers the whole depth of the input, too. Therefore, a feature map $F$ has $K_{width} \cdot K_{height} \cdot depth_{input}$ weights. And since each conv. layer has $depth_{output}$ of these feature maps the total number of weights for a conv. layer is $K_{width} \cdot K_{height} \cdot depth_{input} \cdot depth_{output}$.

Which is $5 \cdot 5 \cdot 100 \cdot 150$ for the second conv. layer in your example.

In case this is still unclear have a look at this explanation from a Stanford class.