1x1 Convolution. How does the math work?

Question

So I stumbled upon Andrew Ng's course on $1x1$ convolutions. There, he explains that you can use a $1x1x192$ convolution to shrink it.

But when I do:

input_ = torch.randn([28, 28, 192])
filter = torch.zeros([1, 1, 192])

out = torch.mul(input_,filter)

I obviously get $28x28x192$ matrix. So how can I shrink it?

Just add the result of every $1x1x192 * 1x1x192$ kernel result? So I'd get a $28x28x1$ matrix?

Answer 1

Let's go back at normal convolution: let's say you have a 28x28x3 image (3 = R,G,B).

I don't use torch, but keras, but the principle applies I think.

When you apply a 2D Convolution, passing the size of the filter, for example 3x3, the framework adapt your filter from 3x3 to 3x3x3! Where the last 3 it's due to the dept of the image.

The same happens when, after a first layer of convolution with 100 filters, you obtain an image of size 28x28x100, at the second convolution layer you decide only the first two dimension of the filter, let's say 4x4. The framework instead, applies a filter of dimension 4x4x100!

So, to reply at your question, if you apply 1x1 convolution to 28x28x100, passing number of filters of k. You obtain an activation map (result) of dimension 28x28xk.

And that's the shrink suggested by Ng.

Again to fully reply to your question, the math is simple, just apply the theory of the convolution using 3D filters. Sum of multiplication of overlapping elements between filter and image.

Edit: Simple example

I am going to show you a simple example of the operations that occur during the deep convolution between M, a tensor of dimension (z= 2, x=2, y=2), where you can see z as your k that you want to shrink to 1 and W, a filter of dimension (2, 1, 1). You will have to implement your own function with a loop to operate the stride of the filter.

import numpy as np

M = np.array([[[1, 2],[3, 4]], [[5, 6],[7, 8]]])
W = np.array([[[2.]],[[3.]]])

C = np.ones(shape=(2, 2))

c[0,0] = np.sum(M[:, 0, 0]*w.T)
c[0,1] = np.sum(M[:, 0, 1]*w.T)
c[1,0] = np.sum(M[:, 1, 0]*w.T)
c[1,1] = np.sum(M[:, 1, 1]*w.T)