Multiple-input multiple-output CNN with custom loss function

Question

I have a set of 2D input arrays $(n\times m)$ namely $A,B,C$ and I would like to predict two 2D output arrays namely $d,e$ for which I have the expected values. You can think of the inputs/outputs as grey images if you like.

Because the spatial information is relevant (there are actually 2D physical domains) I would like to use a convolutional neural network to predict $d$ and $e$. My design (not tested yet) looks as follows:

Because I have multiple inputs, I guess I should use multiple columns (or branches) to find different features for each of the inputs (they look fairly different). Each of these columns follows a encoding-decoding architecture used in segmentation (see SegNet): Conv2D block involves a convolution+batch normalisation+ReLU layer. Deconv2D involves a deconvolution+batch normalisation+ReLU.

Then, I can merge the output of each column by either concatenating, averaging, or taking the maximum for example. To obtain the original $m\times n$ shape for each of the outputs I have seen I could do this with a $1\times1$ kernel convolution.

I would like to predict the two outputs from that single layer. Is that okay from the network structure point of view? Finally, my loss function depends on the outputs themselves compared with the target plus another relation I would like to impose.

I would like to have some expert opinion on this since this is my first design of a CNN and I am not sure if I it makes sense as it is now and/or if there are better approaches (or network architectures) to this problem.

Answer 1

Looks correct. When you use the same branches in a neural network architecture this is called a siamese-type neural network. You can look here for how to implement it in PyPorch. In my opinion, PyTorch has the easiest framework for this task. Also, you can find the implementation of an encoder-decoder network here.

This is a frequently used technique when you need to compute, for example, the distance between a few inputs. But, I have doubts about the concatenation part. Which dimension do you need for outputs? If it is the same as the input, then everything looks fine. If it is a different dimension, then just add a fully connected layer after the last convolution.