Why convolution over volume sums up across channels?

Question

A simple question about convolution over volume .

Say we have an image with dimensions $(n, n, 3)$ and we apply a filter of dimension $(k, k, 3)$ this outputs an matrix of dimension $(n-k+1, n-k+1)$.

Why do we sum across channels in this case. Don't we lose information by mixing different channels. In case of images, this implies mixing information in R, G, B channels? For ex. when trying to detect traffic signal lights, such mixing can be fatal.

Answer 1

CNN filters are used for edge detection only. These edges are basically detected by a mathematical functions and as a result get more and more complex in deeper layers (cascading functions) enabling it to detect complex features.

In your question 2 points need to be noted:

• Traffic lights are not pure red, green or yellow. They may have intensity differences which will get reflected in the image matrix. CNN can capitalise on those differences.
• The weights of CNN filters are randomly initialised, so given you have a large data-set CNN will eventually learn which out of the R,G,B channel is contributing to a specific colour.

Also intuitionally a Traffic Signal is not a light only. It consists of a 3 light device. Consider this, a B/W image is shown to you of a Traffic Light in which a particular colour is on and the colour label is known. You will associate the colour label with the position of the bulb, even though you do not know the colour. Same will happen in a CNN, if the channels have identical values for each colour CNN will learn to recognise from the position of the bulb.

Answer 2

There are multiple points that I try to explain them.

First, each filter for convolutional networks for images is a 3d volume. Consequently, whenever it is said we have $n$ filters, means we have $n$ volumes of those 3d filters.

Second, you can consider each convolutional layer as an MLP which is applied to small regions of the input. These are applied to different regions of the input to investigate whether a typical pattern is in that region or not. These patterns are going to be learned by means of cost functions. You can easily consider that for each filter which is a volume, you are concretely doing a summation over weighted inputs, exactly as MLPs.

Third, as the result, your trained filters will decide to choose the information of which channels depending on the task using the cost function. They may be in a single plane or they may be among multiple of them.