My Understanding of Grouped Convolutions
Let say we have some data with the dimensions [100,100,32]
(lets ignore batch size and assume channels last) and we want to pass it to a convolutional layer with 64 filters. Without grouping, we could pass the input directly into the second convolutional layer and get one output with a shape [100,100,64]
.
Alternatively we can split the input into n groups. For example lets say n=2
. This changes the input from 1 [100,100,32]
tensor to 2 [100,100,16]
tensors. Then we pass each input to a different convolutional layer with 64/n = 32
filters to get two outputs with a shape of [100,100,32]
. These two inputs are than concatenated channel wise to get a single output with a shape of [100,100,64]
.
The Problem
If this is correct, I understand how this can be useful for distributed training on multiple GPUs/CPUs (which I believe is what AlexNet did). However, I have seen claims that grouped convolutions improve performance and I feel that these two approaches (grouped and ungrouped) are mathematically identically and should have no difference in performances.
My Reasoning
Each filter is made up of c kernels; where c is the number of channels in the input. This is because each kernel is applied to a single channel. There is no interaction between kernels in the same filter. Therefore, regardless of how the channels are placed (ie. in one stack or in a groups) the resulting filter will be the same.
Am I mistaken?
Places that claim that grouped convolutions improve performnces