How to align the description of a convolutional neural network in keras with wikipedia's conceptual model?

Question

I was going through the introductory guide to convolutional neural networks in tensor flow here

And I was trying to logically map some of the code I saw to my actual understanding of how convolutional neural networks function.

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))

So when I read this code line by line what I see is

1. Create a model with multiple sequential layers

2. The first layer will take a 32x32 image with 3 channels (a (32,32,3) tensor of numbers) We now for each channel (32 x 32 matrix of numbers), perform a convolution with a (3x3) matrix of numbers resulting in a (32 - 3 + 1, 32- 3 + 1) tensor per channel.

   At this point it is not clear to me what the significance of the 32 is in the line of code

   model.add(layers.Conv2D(32, ... )

   I assume the Relu activation function is applied element-wise to each of the (30,30,3) elements yielding the output for the layer which we then pass to our next layer for pooling

3. So now this (30, 30, 3) tensor gets sent for pooling to result in a (2,2,3) tensor where each (30 x 30) channel was reduced using a maxpool that divided the matrix up into 4 quadrants and took the maximum of each.

And at this point things just go horribly wrong if you try to force this to make sense with the assumptions I have made. The number of outputs of all the convolutional and pooling layers is not longer 64, so its unclear how to match this to the dense layer at the end, and in my experience one does not normally just ignore numbers that especially change midway through the code.

So how do I actually read this code and directly and map it to the architecture of a convolutional network such as the one described in wikipedia by this image:

My (probably incorrect) interpretation of this image is

1. There will be an image as input (with maybe multiple color channels).

2. Each copy of the image will go through 1 or more convolutions and then 1 or more sampling stages where the output tensor of the convolution stage is compatible with the input tensor of the sampling stage,

3. the resultant pool of final samples should have the same number of elements as the first layer of the dense network.

My (possibly incorrect) definition of matrix convolution. Let $A_{mn},B_{op}$ be matrices with dimension $m \times n$ and $o \times p$ then $ A \star B_{u,v} $ is a $(m - o + 1 \times n-p+1)$ matrix whose elements are defined as

$$ A \star B_{u,v} = \sum_{i=0}^{o-1} \sum_{j=0}^{p-1} A_{u+i, v+j} B_{i, j}$$

My (possibly incorrect) definition of downsampling

If A is a $(d,d)$ matrix that you want to MaxPool shrink to a $(u,u)$ matrix called B and $d \equiv 0 \mod u$ then

$$ B_{i,j} = \operatorname{Max}\left\lbrace A_{p,q}| i \frac{d}{u} \le p < i \frac{d}{u} + \frac{d}{u}, j \frac{d}{u} \le q < j \frac{d}{u} + \frac{d}{u} \right\rbrace $$

(In english this reads to compute $B_{i,j}$ just go to the $(i,j)$-block of Matrix A of size $(\frac{d}{u}, \frac{d}{u})$ and compute its maximum element). Here all my indexing starts at 0.

Answer 1

There are two misunderstandings:

1. How Convolution with layers works
2. How the pooling is defined

Convolution with layers

Let's start with the ominous 32:

model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))

From the API docs, we get that this is the filter parameter with the following description:

Integer, the dimensionality of the output space (i.e. the number of output filters in the convolution).

Basically, this means that the output of of this first layer contains 32 channels. So the layer transforms a (32, 32, 3) - Tensor into a (30, 30, 32) - Tensor. Which means, that the math behind the convolution needs more parameters and indices:

• $A$ is a $n\times m\times c$ - Tensor, where $c=3$ is the number of input channels
• $B$ is a $o\times p\times c\times d$ - Tensor, wehere $d=32$ is the number of output channels.
• The convolution is given by: $$(A\star B)_{u,v,w}=\sum_{i=0}^{o-1}\sum_{j=0}^{p-1}\sum_{k=0}^{c-1} A_{u+i,v+j,k}B_{i,j,k,w}$$

Definition of pooling

Again, a glance into the API docs reveals, that (2,2) is the pool_size which is explained as:

integer or tuple of 2 integers, window size over which to take the maximum. (2, 2) will take the max value over a 2x2 pooling window. If only one integer is specified, the same window length will be used for both dimensions.

This means, that a block of (2,2) pixels is pooled into one pixel. Hence, the pooling layer cuts the side length in half, i.e. a (30,30,32) - Image into a (15,15,32) - Image