Combining 2 Neural Networks

2 images as input, x1 and x2 and try to use convolution as a similarity measure. The idea is that the learned weights substitute more traditional measure of similarity (cross correlation, NN, ...). Defining my forward function as follows:

def forward(self,x1,x2):
out_conv1a = self.conv1(x1)
out_conv2a = self.conv2(out_conv1a)
out_conv3a = self.conv3(out_conv2a)

out_conv1b = self.conv1(x2)
out_conv2b = self.conv2(out_conv1b)
out_conv3b = self.conv3(out_conv2b)


Now for the similarity measure:

out_cat = torch.cat([out_conv3a, out_conv3b],dim=1)
futher_conv = nn.Conv2d(out_cat)


Question is as follows:

1. Would Depthwise/Separable Convolutions as in the google paper yield any advantage over 2d convolution of the concatenated input. For that matter can convolution be a similarity measure, cross correlation and convolution are very similar.

2. It is my understanding that the groups=2 option in conv2d would provide 2 separate inputs to train weights with, in this case each of the previous networks weights. How are these combined afterwards?

• IMHO depthwise/separable convolutions provide marginal improvement and only in specific areas. IMHO ResNet is a better start overall then google's neural networks. And in my head convolutions and cross correlation are completely different things with completely different math behind. You can calculate similarity score based on the output of the layer before the last one. And weights could be completely different for the same results. So IMHO weights are not a good approximation of similarity. Oct 13 '18 at 22:27
• Before we consider all of those details, how are you going to train your network? What's your target? Jan 7 '19 at 1:30

2. Depthwise separable conv is an extreme case of grouped conv, where num_groups=num_input_channels. Setting number of groups to 2 means slicing the input into two inputs across the channel dim and learning two separate filters for each sliced input. Here is nice tutorial on that. The output is combined by stacking each group's output. Often this is followed by 1x1 conv to mix the data flow between the groups.