I'm training a model that has two neural networks. One of them is a resnet18 CNN which has as it's input images. The other one is a small one hidden layer network that has as it's input four other variables.
At this moment I concatenate the outputs of these networks in the first (and only) fully connected layer, after which datapoints are classified into three classes in the classification layer.
I was wondering how I could design my network in such a way that the two different neural networks have a specific amount of influence on the classification. For instance, I'd like the CNN to have 60% 'say'/influence in the eventual classification, and have the other NN have 40% 'say'/influence.
I can't find an intuitive way to to this in the current structure. The only way I can imagine doing it is by separating the networks, and then weighting the losses of both the networks. Does anybody know another way to achieve this without separating the networks?