My Understanding of Grouped Convolutions
Let say we have some data with the dimensions
[100,100,32] (lets ignore batch size and assume channels last) and we want to pass it to a convolutional layer with 64 filters. Without grouping, we could pass the input directly into the second convolutional layer and get one output with a shape
Alternatively we can split the input into n groups. For example lets say
n=2. This changes the input from 1
[100,100,32] tensor to 2
[100,100,16] tensors. Then we pass each input to a different convolutional layer with
64/n = 32 filters to get two outputs with a shape of
[100,100,32]. These two inputs are than concatenated channel wise to get a single output with a shape of
If this is correct, I understand how this can be useful for distributed training on multiple GPUs/CPUs (which I believe is what AlexNet did). However, I have seen claims that grouped convolutions improve performance and I feel that these two approaches (grouped and ungrouped) are mathematically identically and should have no difference in performances.
Each filter is made up of c kernels; where c is the number of channels in the input. This is because each kernel is applied to a single channel. There is no interaction between kernels in the same filter. Therefore, regardless of how the channels are placed (ie. in one stack or in a groups) the resulting filter will be the same.
Am I mistaken?
Places that claim that grouped convolutions improve performnces