# Multiple-input multiple-output CNN with custom loss function

I have a set of 2D input arrays $$(n\times m)$$ namely $$A,B,C$$ and I would like to predict two 2D output arrays namely $$d,e$$ for which I have the expected values. You can think of the inputs/outputs as grey images if you like.

Because the spatial information is relevant (there are actually 2D physical domains) I would like to use a convolutional neural network to predict $$d$$ and $$e$$. My design (not tested yet) looks as follows:

Because I have multiple inputs, I guess I should use multiple columns (or branches) to find different features for each of the inputs (they look fairly different). Each of these columns follows a encoding-decoding architecture used in segmentation (see SegNet): Conv2D block involves a convolution+batch normalisation+ReLU layer. Deconv2D involves a deconvolution+batch normalisation+ReLU.

Then, I can merge the output of each column by either concatenating, averaging, or taking the maximum for example. To obtain the original $$m\times n$$ shape for each of the outputs I have seen I could do this with a $$1\times1$$ kernel convolution.

I would like to predict the two outputs from that single layer. Is that okay from the network structure point of view? Finally, my loss function depends on the outputs themselves compared with the target plus another relation I would like to impose.

I would like to have some expert opinion on this since this is my first design of a CNN and I am not sure if I it makes sense as it is now and/or if there are better approaches (or network architectures) to this problem.