Channels in convolutional layer

I usually see convolutions performed over all the channels of the input. For example a $$3x3$$ kernel is really a $$3x3xN$$ kernel for a an input with $$N$$ channels, thus resulting in a single output channel per filter.

What would happen, if you were to do a convolution on individual channels or pairs of channels instead, with independent filters per channel? I am wondering if when certain channels are not correlated could you prevent some information loss by keeping the channels separate for some layers?