How to calculate convolution for 2nd conv Layer in CNN, Do we need to average across all feature maps?

Question

I understand that for the first layer (assuming we have a grayscale image) we calculate the convolution of 3*3 receptive field as a weighted sum of receptive weights with pixels

$ x1 · w1 + x2 · w2 + x3·w3 + ... + xn · wn$

But for the second layer, where we already have $N$ feature maps in the last convolution layer, how would we calculate the convolution(for a particular pixel/cell)? should we take an average of the $N$ weighted sums we have from feature maps?

If the question isn't clear, I have tried to highlighted it from the image taken from famous 3d visualization.

In the image, for the pixel in question(hovered and squared) we have inputs coming from 4 feature maps. I think they are four integers (weighted sums of the receptive field of the bottom right corner from each feature map).

How is the value (convolution) for this particular pixel/cell calculated? showed we take average like below?

(I can add more details if the question is not clear)

weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 / 4

Answer 1

No, you don't average across all feature maps.

When the input has multiple channels, you need your convolution filter to have the same number of channels. Therefore, the filter "covers" the full depth of the input. Then, you simply perform the element-wise multiplication of the filter with the overlapping region in the input and add all the resulting elements together.

Therefore, if the input to a convolutional layer is a grayscale image, a filter is of dimensions [3,3,1], while if the input is a color image (with 3 channels), your filter is [3, 3, 3], like this (source):

The same happens with feature maps with N channels: we need a filter with N channels.