How are the channels handled in CNN? Is it independently processed or fused?

Question

Let's assume that we are talking about 2D convolutions applied on images.

In a grayscale image, the data is a matrix of dimensions $w \times h$, where $w$ is the width of the image and $h$ is its height. In a color image, we normally have 3 channels: red, green and blue; this way, a color image can be represented as a matrix of dimensions $w \times h \times c$, where $c$ is the number of channels, that is, 3.

A convolution layer receives the image ($w \times h \times c$) as input, and generates as output an activation map of dimensions $w' \times h' \times c'$. The number of input channels in the convolution is $c$, while the number of output channels is $c'$.

My confusion is that will CNN operate on the fused representation of the data if $c =2$ or 3 or 4 etc? Or does it operate on each channel at a time and then stacks the results? Say I have 4 channels each channel is a 2D matrix then would CNN internally form a fusion of the 4 channels and make some sort of a representation?

Answer 1

Let $n$ be a convolutional layer with dimensions $w' \times h' \times c'$. Then each of its $c'$ filters is connected to all $c$ filters (or channels*) of the previous layer.

I find it helpful to look at the number of weights here: A single filter of that convolutional layer $n$ with kernel size $k'\times k'$ will have $c \times k' \times k'$ weights. And since layer $n$ has $c'$ of these filters it has a total of $c \times k' \times k' \times c'$ weights.

In a toy example with a 3 channel input layer followed by a conv. layer with 5 channels this would look as follows (not showing the bias here for simplicity):

As you can see from the drawing each feature map of the conv. layer receives all input channels as an input (and the same would apply if this was not an input layer but a conv. layer with 3 feature maps).

---

*Note that it does not make a difference whether the previous layer is a conv. layer too or the input layer - in the first case you call its depth "filters" and in the second you call it "channels" but that does not change how it is connected to the following conv. layer.