How to calculate convolution for 2nd conv Layer in CNN, Do we need to average across all feature maps?

I understand that for the first layer (assuming we have a grayscale image) we calculate the convolution of 3*3 receptive field as a weighted sum of receptive weights with pixels

$$x1 · w1 + x2 · w2 + x3·w3 + ... + xn · wn$$

But for the second layer, where we already have $$N$$ feature maps in the last convolution layer, how would we calculate the convolution(for a particular pixel/cell)? should we take an average of the $$N$$ weighted sums we have from feature maps?

If the question isn't clear, I have tried to highlighted it from the image taken from famous 3d visualization.

In the image, for the pixel in question(hovered and squared) we have inputs coming from 4 feature maps. I think they are four integers (weighted sums of the receptive field of the bottom right corner from each feature map).

How is the value (convolution) for this particular pixel/cell calculated? showed we take average like below?

(I can add more details if the question is not clear)

weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 / 4

• Are you referring to Average Pooling? See this blog on Pooling. Commented Apr 9, 2021 at 6:56

Therefore, if the input to a convolutional layer is a grayscale image, a filter is of dimensions [3,3,1], while if the input is a color image (with 3 channels), your filter is [3, 3, 3], like this (source):