I understand that for the first layer (assuming we have a grayscale image) we calculate the convolution of 3*3 receptive field as a weighted sum of receptive weights with pixels
$ x1 · w1 + x2 · w2 + x3·w3 + ... + xn · wn$
But for the second layer, where we already have $N$ feature maps in the last convolution layer, how would we calculate the convolution(for a particular pixel/cell)? should we take an average of the $N$ weighted sums we have from feature maps?
If the question isn't clear, I have tried to highlighted it from the image taken from famous 3d visualization.
In the image, for the pixel in question(hovered and squared) we have inputs coming from 4 feature maps. I think they are four integers (weighted sums of the receptive field of the bottom right corner from each feature map).
How is the value (convolution) for this particular pixel/cell calculated? showed we take average like below?
(I can add more details if the question is not clear)
weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 / 4