1x1 Convolution. How does the math work?

So I stumbled upon Andrew Ng's course on $$1x1$$ convolutions. There, he explains that you can use a $$1x1x192$$ convolution to shrink it.

But when I do:

input_ = torch.randn([28, 28, 192])
filter = torch.zeros([1, 1, 192])

out = torch.mul(input_,filter)

I obviously get $$28x28x192$$ matrix. So how can I shrink it?

Just add the result of every $$1x1x192 * 1x1x192$$ kernel result? So I'd get a $$28x28x1$$ matrix?

Let's go back at normal convolution: let's say you have a 28x28x3 image (3 = R,G,B).

I don't use torch, but keras, but the principle applies I think.

When you apply a 2D Convolution, passing the size of the filter, for example 3x3, the framework adapt your filter from 3x3 to 3x3x3! Where the last 3 it's due to the dept of the image.

The same happens when, after a first layer of convolution with 100 filters, you obtain an image of size 28x28x100, at the second convolution layer you decide only the first two dimension of the filter, let's say 4x4. The framework instead, applies a filter of dimension 4x4x100!

So, to reply at your question, if you apply 1x1 convolution to 28x28x100, passing number of filters of k. You obtain an activation map (result) of dimension 28x28xk.

And that's the shrink suggested by Ng.

Again to fully reply to your question, the math is simple, just apply the theory of the convolution using 3D filters. Sum of multiplication of overlapping elements between filter and image.

Edit: Simple example

I am going to show you a simple example of the operations that occur during the deep convolution between M, a tensor of dimension (z= 2, x=2, y=2), where you can see z as your k that you want to shrink to 1 and W, a filter of dimension (2, 1, 1). You will have to implement your own function with a loop to operate the stride of the filter.

import numpy as np

M = np.array([[[1, 2],[3, 4]], [[5, 6],[7, 8]]])
W = np.array([[[2.]],[[3.]]])

C = np.ones(shape=(2, 2))

c[0,0] = np.sum(M[:, 0, 0]*w.T)
c[0,1] = np.sum(M[:, 0, 1]*w.T)
c[1,0] = np.sum(M[:, 1, 0]*w.T)
c[1,1] = np.sum(M[:, 1, 1]*w.T) • I'm trying to achive this with matrix multiplication. And there is noway I can get to 28x28xK what's the calculation that shrinks it? If you look at @André answer, then what's g? – Mihkel L. Sep 23 '18 at 7:46
• You get the shrinking because the result of convolution between overlapping regions between a tensor of depth k and a filter of depth k is a matrix with depth 1! For each region you get a scalar, and all together they form a matrix. So you shrink from k to 1! To obtain a result of depth X, you just stack the result of X filters. – Francesco Pegoraro Sep 23 '18 at 14:07
• @Francesco Pegoraro Show me some real calculation. I have matrix 28x28x192. When I multiply it with 1x1x192 then i get 28x28x192. No Scalar. Show me how I get scalar please. – Mihkel L. Sep 23 '18 at 18:30
• best thing would be to show PyTorch code that compiles W x W x K matrix into W x W matrix :) – Mihkel L. Sep 23 '18 at 18:40
• I am going to modify my answer with an example in numpy – Francesco Pegoraro Sep 23 '18 at 22:17