3
$\begingroup$

The example below is taken from the lectures in deeplearning.ai shows that the result is the sum of the element-by-element product (or "element-wise multiplication". The red numbers represent the weights in the filter:

$(1*1)+(1*0)+(1*1)+(0*0)+(1*1)+(1*0)+(0*1)+(0*0)+(1*1) = 1+0+1+0+1+0+0+0+1 = 4 $

enter image description here

HOWEVER, most resources say that it's the dot product that's used:

"…we can re-express the output of the neuron as , where is the bias term. In other words, we can compute the output by y=f(x*w) where b is the bias term. In other words, we can compute the output by performing the dot product of the input and weight vectors, adding in the bias term to produce the logit, and then applying the transformation function."

Buduma, Nikhil; Locascio, Nicholas. Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms (p. 8). O'Reilly Media. Kindle Edition.

"We take the 5*5*3 filter and slide it over the complete image and along the way take the dot product between the filter and chunks of the input image. For every dot product taken, the result is a scalar."

https://medium.com/technologymadeeasy/the-best-explanation-of-convolutional-neural-networks-on-the-internet-fbb8b1ad5df8

"Each neuron receives some inputs, performs a dot product and optionally follows it with a non-linearity."

http://cs231n.github.io/convolutional-networks/

"The result of a convolution is now equivalent to performing one large matrix multiply np.dot(W_row, X_col), which evaluates the dot product between every filter and every receptive field location."

http://cs231n.github.io/convolutional-networks/

However, when I research how to compute the dot product of matrics, it seems that the dot product is not the same as summing the element-by-element multiplication. What operation is actually used (element-by-element multiplication or the dot product?) and what is the primary difference?

$\endgroup$
2
$\begingroup$

The convolution can be written as

$I(x,y) = \sum_{i=0}^n \sum_{j=0}^m k(i,j) I(x+i, y+j)$

where the kernel $k$ is of size $n$ by $m$. The image is size $N$ by $M$. Ok so in your example we are evaluating $I(0,0)$, our kernel is $n=3$ and $m=3$. Then our image is $N=5$ and $M=5$. If you go through the equation then you will see that you get exactly the result you showed.

$(1∗1)+(1∗0)+(1∗1)+(0∗0)+(1∗1)+(1∗0)+(0∗1)+(0∗0)+(1∗1)=1+0+1+0+1+0+0+0+1=4$

This is a dot product as defined as being a sum of products. We multiply element-wise and then we add them all together. The result is the answer you got $4$.

The entire convolution operation across the image is a matrix where each entry is this dot product.

$\endgroup$
  • 1
    $\begingroup$ The real name of this operation is called cross-correlation in mathematics. Because in CNNs we don't mirror the signal, it is not convolution, although it is called convolution! :D $\endgroup$ – Media Mar 19 '18 at 6:21
2
$\begingroup$

I believe the key is that when the filter is convolving some part of the image (the "receptive field") each number in the filter (i.e. each weight) is first flattened into vector format. Likewise, the pixels of the image are also flattened into vector format. THEN, the dot product is calculated. Which is the exact same thing as finding the sum of the element-by-element (element-wise) multiplication.

Of course, these flattened vectors can also be combined in a matrix format, as the below image shows. In this case then true matrix multiplication can be used, but it's important to note that the flattening the image pixels from each convolution and also the weights filter is the precursor.

enter image description here

image credit: TensorFlow and Deep Learning without a PhD, Part 1 (Google Cloud Next '17)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.