For a sub problem I need a network that produces a single continuous number $y$, however during training I have more information available about the output, the output is a weighted sum between a 4 other numbers $y_{1..4}$, the weights are known and constant. All of $y_{1..4}$ are dependent on the input $x$ and I feel like incorporating this knowledge into the network architecture might be beneficial for training. My current idea is to have a number of shared layers, then have 4 individual intermediate layers that all regress to the specific $y_{1..4}$ and then have $y=\sum_{i=1}^4a_iy_i$, with $a_i$ being the weight for $y_i$.
My question is about a good loss function, I'm interested in regressing the mean so using $L_2$ loss makes sense but only doing it at the end doesn't incorporate the prior knowledge. You could add all the $L_2$ losses for the individual $y_i$, they should probably be weighted by how much weight they have in the total sum. However then we don't directly optimize $y$, sometimes the network could be more biased than normal by making a mistake in the same direction for all $y_{1..4}$. My idea is now to do it on the individual $y_i$ but also on $y$, weighing the sum of the $L_2$ of the individual $y_i$ equally to $L_2$ of $y$. The weight between these two parts of the loss function could be different and I guess is a hyperparameter of the model. My question is if there are any papers that have worked with a similar idea to see which of the options work better or if there is another approach to do this. Splitting the value and advantage streames in deep reinforcement learning is what inspired me to look at this.
EDIT: The goal is indeed to get the lowest $L_2$ loss on the final $y$, but I feel like the network should be able to learn better because I have more information with regards to how $y$ is built up. My question is how to best incorporate this additional information, I know I can test some different scenarios but was wondering if there is any literature about similar approaches before implementing things that end up inferior.
