In a convolutional neural network (CNN), when convolving the image, is the operation used the dot product or the sum of element-wise multiplication?

Question

The example below is taken from the lectures in deeplearning.ai shows that the result is the sum of the element-by-element product (or "element-wise multiplication". The red numbers represent the weights in the filter:

$(1*1)+(1*0)+(1*1)+(0*0)+(1*1)+(1*0)+(0*1)+(0*0)+(1*1) = 1+0+1+0+1+0+0+0+1 = 4 $

HOWEVER, most resources say that it's the dot product that's used:

"…we can re-express the output of the neuron as , where is the bias term. In other words, we can compute the output by y=f(x*w) where b is the bias term. In other words, we can compute the output by performing the dot product of the input and weight vectors, adding in the bias term to produce the logit, and then applying the transformation function."

Buduma, Nikhil; Locascio, Nicholas. Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms (p. 8). O'Reilly Media. Kindle Edition.

"We take the 553 filter and slide it over the complete image and along the way take the dot product between the filter and chunks of the input image. For every dot product taken, the result is a scalar."

The best explanation of Convolutional Neural Networks on the Internet!

"Each neuron receives some inputs, performs a dot product and optionally follows it with a non-linearity."

CS231n Convolutional Neural Networks for Visual Recognition

"The result of a convolution is now equivalent to performing one large matrix multiply np.dot(W_row, X_col), which evaluates the dot product between every filter and every receptive field location."

CS231n Convolutional Neural Networks for Visual Recognition

However, when I research how to compute the dot product of matrics, it seems that the dot product is not the same as summing the element-by-element multiplication. What operation is actually used (element-by-element multiplication or the dot product?) and what is the primary difference?

Answer 1

The convolution can be written as

$I(x,y) = \sum_{i=0}^n \sum_{j=0}^m k(i,j) I(x+i, y+j)$

where the kernel $k$ is of size $n$ by $m$. The image is size $N$ by $M$. Ok so in your example we are evaluating $I(0,0)$, our kernel is $n=3$ and $m=3$. Then our image is $N=5$ and $M=5$. If you go through the equation then you will see that you get exactly the result you showed.

$(1∗1)+(1∗0)+(1∗1)+(0∗0)+(1∗1)+(1∗0)+(0∗1)+(0∗0)+(1∗1)=1+0+1+0+1+0+0+0+1=4$

This is a dot product as defined as being a sum of products. We multiply element-wise and then we add them all together. The result is the answer you got $4$.

The entire convolution operation across the image is a matrix where each entry is this dot product.

Answer 2

I believe the key is that when the filter is convolving some part of the image (the "receptive field") each number in the filter (i.e. each weight) is first flattened into vector format. Likewise, the pixels of the image are also flattened into vector format. THEN, the dot product is calculated. Which is the exact same thing as finding the sum of the element-by-element (element-wise) multiplication.

Of course, these flattened vectors can also be combined in a matrix format, as the below image shows. In this case then true matrix multiplication can be used, but it's important to note that the flattening the image pixels from each convolution and also the weights filter is the precursor.

image credit: TensorFlow and Deep Learning without a PhD, Part 1 (Google Cloud Next '17)