Channels in convolutional layer

Question

I usually see convolutions performed over all the channels of the input. For example a $3x3$ kernel is really a $3x3xN$ kernel for a an input with $N$ channels, thus resulting in a single output channel per filter.

What would happen, if you were to do a convolution on individual channels or pairs of channels instead, with independent filters per channel? I am wondering if when certain channels are not correlated could you prevent some information loss by keeping the channels separate for some layers?

Answer 1

The technique you have mentioned is called "Depth-wise Separable Convolutions" and it relies on the idea that spatial and depth-wise information can be decoupled. Its main advantage is that compared with standard convolution, since it doesn't need to perform convolution across all channels, it has considerably less connections, (so less parameters) which results in a lighter model.

There is an architecture named "Xception" ("Extreme version of Inception") by Google that uses a version of such convolutional layers and produces close to state-of-the-art results. There is a really good review of it at Towards Data Science.

Personally I don't like the idea of decoupling the channels, because I think it gets us a less "sophisticated" model, but it seems to work so who am I to judge?