0
$\begingroup$

My Understanding of Grouped Convolutions

Let say we have some data with the dimensions [100,100,32] (lets ignore batch size and assume channels last) and we want to pass it to a convolutional layer with 64 filters. Without grouping, we could pass the input directly into the second convolutional layer and get one output with a shape [100,100,64].

Alternatively we can split the input into n groups. For example lets say n=2. This changes the input from 1 [100,100,32] tensor to 2 [100,100,16] tensors. Then we pass each input to a different convolutional layer with 64/n = 32 filters to get two outputs with a shape of [100,100,32]. These two inputs are than concatenated channel wise to get a single output with a shape of [100,100,64].

The Problem

If this is correct, I understand how this can be useful for distributed training on multiple GPUs/CPUs (which I believe is what AlexNet did). However, I have seen claims that grouped convolutions improve performance and I feel that these two approaches (grouped and ungrouped) are mathematically identically and should have no difference in performances.

My Reasoning

Each filter is made up of c kernels; where c is the number of channels in the input. This is because each kernel is applied to a single channel. There is no interaction between kernels in the same filter. Therefore, regardless of how the channels are placed (ie. in one stack or in a groups) the resulting filter will be the same.

Am I mistaken?

Places that claim that grouped convolutions improve performnces

$\endgroup$

1 Answer 1

0
$\begingroup$

As mentioned in the article also, the different groups learn different representations for the data.

In a normal convolutional network, each layer learns a unique representation. But, here, in the same layer, we are able to derive different representations. It can also related to the software engineering principle of Separation of Concerns. Since, different filter groups are trained separately, they are bound to learn things differently.

From a statistical perspective, when a traditional convolutional layers is trained, the correlation between the kernel weights will be there. Since, here we have separated them into groups, the correlation won't happen since they are trained separately.

$\endgroup$
5
  • $\begingroup$ I get that it show a unique VISUAL representation but if you don't use grouping you could just ultimately group the filters and get the same thing (or at least that is where my confusion lies). Also, doesn't the correlation between kernel's arrive because correlated weights better extract features and that minimizes the loss. But mathematically grouped convolutions are the same as ungrouped convolutions and should not have and impact on the loss. $\endgroup$
    – J Houseman
    Jul 20, 2021 at 12:59
  • $\begingroup$ I tried testing group convolutions on a simple Fashion-MNIST example. I ran grouped and un grouped versions of the architecture for 10 epochs for a total of four time. There was never more than a 0.3% difference between the two in accuracy and validation accuracy. Although, grouped convolutions performed between in 3 of the 4 instances. I don't think this shows grouped convolutions being better than ungrouped convolutions. $\endgroup$
    – J Houseman
    Jul 20, 2021 at 13:01
  • $\begingroup$ Looking at more paper's I feel the literature is mixed on this topic. One paper I saw says that grouped convolutions perform just as well as normal convolutions but are more efficent. $\endgroup$
    – J Houseman
    Jul 20, 2021 at 13:05
  • $\begingroup$ You have to just understand the regularizing effect grouped convolutions bring. That is the reason for the accuracy increase. Obviously, it won't be astronomical. $\endgroup$ Jul 20, 2021 at 17:28
  • $\begingroup$ @AbhishekVerma Can you explain what is the "correlation between the kernel weights"? $\endgroup$
    – ado sar
    Jun 4 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.