Does a Convolutional Layer in a Neural Network learn the correlation between its input signals via its kernel?

Question

I am interested in the theory behing what a convolutional neural network learns with its convolutional operations. I think it learns (useful) kernels which measure the correlation between its input signals. Moreover, on a technical perspective: the convolution operation is implemented as "cross-correlation". So, is my assumption right?

Answer 1

In a convolutional neural network (CNN), a convolutional layer has several channels, each of which has one convolution kernel, often written down as a matrix. This convolution kernel is nothing more than a collection of weights used to compute a linear combination of elements of the input.

While both "traditional" dense layers and convolutional layers compute a linear combination of their inputs, convolutional layers have the added structure of preserving spatial information. Additionally, the output of a convolutional layer, called a feature map, can be understood as the result of sliding the kernel along the input.

For instance, take a simple kernel $K = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix}$ and a small input "image" $I = \begin{bmatrix} 9 & 8 & 7 & 6 \\ 5 & 4 & 3 & 2 \\ 1 & 2 & 3 & 4 \\ 5 & 6 & 7 & 8 \end{bmatrix}$. We take an input with only one channel for convenience: if it's the first layer of the CNN, we can assume the values of $I$ represent a grayscale image. Let us look at how the convolution is performed if no padding is added to the image, in which case the output will be a 2 by 2 matrix.

Starting at the top left corner, we convolve $K = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix}$ with $\begin{bmatrix} 9 & 8 & 7 \\ 5 & 4 & 3 \\ 1 & 2 & 3 \end{bmatrix}$. To do so, we multiply the two matrices element-wise, then sum up all the elements of the resulting matrix: $1 \cdot 9 + 2 \cdot 8 + 3 \cdot 7 + 4 \cdot 5 + 5 \cdot 4 + 6 \cdot 3 + 7 \cdot 1 + 8 \cdot 1 + 9 \cdot 3 = 146$.

Doing the same three more times yields the following feature map: $\begin{bmatrix} 146 & 157 \\ 200 & 233 \end{bmatrix}$.

Depending on the kernel of a layer, the feature map will look different, and the information extracted from the input has a different "meaning". The idea behind a CNN is to make the coefficients of the kernel learnable parameters. Often, multiple "meanings" must be extracted from a single input, so a convolutional layer has multiple channels.

However, CNNs didn't invent convolution, they merely made the kernels learnable. Indeed, such methods have been used in "traditional" computer vision (CV) for a long time. In traditional CV, the kernels are hand-crafted to fill specific roles, extract specific types of information.

For edge detection or line detection, you might use a Prewitt filter which is the discrete equivalent of a derivative operator or a slightly more complicated Sobel filter.

In fact, it is not uncommon for a CNN to learn approximations of well-known hand-crafted convolution kernels from traditional CV on the first layer. Also, by visualizing feature maps on deeper layers, as the training of a CNN progresses, gives you good insight into what the specific convolution channels are learning to detect.