For a sub problem I need a network that produces a single continuous number $y$, however during training I have more information available about the output, the output is a weighted sum between a 4 other numbers $y_{1..4}$, the weights are known and constant. All of $y_{1..4}$ are dependent on the input $x$ and I feel like incorporating this knowledge into the network architecture might be beneficial for training. My current idea is to have a number of shared layers, then have 4 individual intermediate layers that all regress to the specific $y_{1..4}$ and then have $y=\sum_{i=1}^4a_iy_i$, with $a_i$ being the weight for $y_i$.

My question is about a good loss function, I'm interested in regressing the mean so using $L_2$ loss makes sense but only doing it at the end doesn't incorporate the prior knowledge. You could add all the $L_2$ losses for the individual $y_i$, they should probably be weighted by how much weight they have in the total sum. However then we don't directly optimize $y$, sometimes the network could be more biased than normal by making a mistake in the same direction for all $y_{1..4}$. My idea is now to do it on the individual $y_i$ but also on $y$, weighing the sum of the $L_2$ of the individual $y_i$ equally to $L_2$ of $y$. The weight between these two parts of the loss function could be different and I guess is a hyperparameter of the model. My question is if there are any papers that have worked with a similar idea to see which of the options work better or if there is another approach to do this. Splitting the value and advantage streames in deep reinforcement learning is what inspired me to look at this.

EDIT: The goal is indeed to get the lowest $L_2$ loss on the final $y$, but I feel like the network should be able to learn better because I have more information with regards to how $y$ is built up. My question is how to best incorporate this additional information, I know I can test some different scenarios but was wondering if there is any literature about similar approaches before implementing things that end up inferior.

  • $\begingroup$ Yes thank you, I was not very clear in my post. The goal is to predict $y$ as good as possible (ie L2), my question was about how to incorporate the prior information better, and for literature about this. $\endgroup$ May 23, 2017 at 9:58
  • $\begingroup$ 1. Are $y_1,..,y_4$ known? Are they known for instances in the training set? Are they known for instances in the test set? Can you edit the question to clarify? 2. Also, it's better not to use "EDIT:" here. Instead, edit the question to read well for someone who encounters it for the first time. There's no need to mark what has changed -- we have built-in revision history for that (click 'edited' under the question to see what has changed). Thank you! $\endgroup$
    – D.W.
    May 23, 2017 at 21:09
  • $\begingroup$ 3. What specifically is your question? 4. You say you want your loss function to be L_2 loss on $y$. So why don't you just make that your loss function? What is the point of all the stuff you mention in the second paragraph? I don't follow you. $\endgroup$
    – D.W.
    May 23, 2017 at 21:10


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.