I understand that for the first layer (assuming we have a grayscale image) we calculate the convolution of 3*3 receptive field as a weighted sum of receptive weights with pixels

$ x1 · w1 + x2 · w2 + x3·w3 + ... + xn · wn$

But for the second layer, where we already have $N$ feature maps in the last convolution layer, how would we calculate the convolution(for a particular pixel/cell)? should we take an average of the $N$ weighted sums we have from feature maps?

If the question isn't clear, I have tried to highlighted it from the image taken from famous 3d visualization.

2nd layer conv

In the image, for the pixel in question(hovered and squared) we have inputs coming from 4 feature maps. I think they are four integers (weighted sums of the receptive field of the bottom right corner from each feature map).

How is the value (convolution) for this particular pixel/cell calculated? showed we take average like below?

(I can add more details if the question is not clear)

weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 / 4
  • $\begingroup$ Are you referring to Average Pooling? See this blog on Pooling. $\endgroup$ Commented Apr 9, 2021 at 6:56

1 Answer 1


No, you don't average across all feature maps.

When the input has multiple channels, you need your convolution filter to have the same number of channels. Therefore, the filter "covers" the full depth of the input. Then, you simply perform the element-wise multiplication of the filter with the overlapping region in the input and add all the resulting elements together.

Therefore, if the input to a convolutional layer is a grayscale image, a filter is of dimensions [3,3,1], while if the input is a color image (with 3 channels), your filter is [3, 3, 3], like this (source):

enter image description here

The same happens with feature maps with N channels: we need a filter with N channels.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.