# Do Convolution Layers in a CNN Treat the Previous Layer Outputs as Channels?

Lets say you have a max pooling layer that gives 10 downsampled feature maps. Do you stack those feature maps, treat them as channels and convolve that 'single image' of depth 10 with a 3d kernel of depth 10? That is how I have generally thought about it. Is that correct?

This visualization confused me: http://scs.ryerson.ca/~aharley/vis/conv/flat.html

On the second convolution layer in the above visualization most of the feature maps only connect to 3 or 4 of the previous layers maps. Can anyone help me understand this better?

Related side question: If our input is a color image our first convolution kernel will be 3D. This means we learn different weights for each color channel (I assume we aren't learning a single 2D kernel that is duplicated on each channel, correct)?

• It seems like this is some non-standard architecture; it is convolving several 3d kernels of different sizes with the first downsampled layer. Is there a description of this architecture somewhere? Nov 11, 2017 at 1:50
• Is what I describe the typical case then? Here is a link to the paper. Though I am mostly just wanting to understand what typically occurs. Nov 11, 2017 at 1:55
• That paper is kind of weird; it describes two different networks, neither of which seems to pertain to the network being displayed. Nov 11, 2017 at 2:03

Yes. The usual convention in a CNN is that each kernel is always the same depth as the input, so you can also think of this as a "stack" of 2D kernels that are associated with the input channels and summed to make one output channel - because under the convention that $N_{in\_channels} = N_{kernel\_depth}$ this is mathematically the same. Expressing as a 3D convolution allows for simpler notation and code.