Combining 2 Neural Networks

Question

2 images as input, x1 and x2 and try to use convolution as a similarity measure. The idea is that the learned weights substitute more traditional measure of similarity (cross correlation, NN, ...). Defining my forward function as follows:

def forward(self,x1,x2):
    out_conv1a = self.conv1(x1)
    out_conv2a = self.conv2(out_conv1a)
    out_conv3a = self.conv3(out_conv2a)

    out_conv1b = self.conv1(x2)
    out_conv2b = self.conv2(out_conv1b)
    out_conv3b = self.conv3(out_conv2b)

Now for the similarity measure:

out_cat = torch.cat([out_conv3a, out_conv3b],dim=1)
futher_conv = nn.Conv2d(out_cat)

Question is as follows:

1. Would Depthwise/Separable Convolutions as in the google paper yield any advantage over 2d convolution of the concatenated input. For that matter can convolution be a similarity measure, cross correlation and convolution are very similar.

2. It is my understanding that the groups=2 option in conv2d would provide 2 separate inputs to train weights with, in this case each of the previous networks weights. How are these combined afterwards?

Answer 1

The paper explicitly states the following lines:

Our work is one more point on a significant trendline started with Xception and MobileNets, that indicates that in any convolutional model, whether for 1D or 2D data, it is possible to replace convolutions with depthwise separable convolutions and obtain a model that is simultaneously cheaper to run, smaller, and performs a few percentage points better.

A Convolutional neural network forms a chain of differentiable feature learning modules, structured as a discrete set of units, each trained to learn a particular feature. If trained and reused, these could be extended to be used for cosine similarity findings. Depthwise separable convolutions define individual feature paths and so it would be easy for this case where you have concatenated the outputs.

Answer 2

1. Depthwise separable convolution followed by 1x1 convolution is an alternative to the convenient 2d convolution. They were introduced in https://arxiv.org/abs/1704.04861 to aiming on reducing the computational effort. The combination of depthwise and pointwise convolutions often learn pretty much the same representation as the normal conv.

2. Depthwise separable conv is an extreme case of grouped conv, where num_groups=num_input_channels. Setting number of groups to 2 means slicing the input into two inputs across the channel dim and learning two separate filters for each sliced input. Here is nice tutorial on that. The output is combined by stacking each group's output. Often this is followed by 1x1 conv to mix the data flow between the groups.