I'm training a convolutional neural network with 1 input, and 3 outputs: a classification and 2 regression outputs.
I initially used 3 separate models, but I found that combining the 1st and 2nd loss functions into 1 model improves accuracy/rmse for both. I'm applying multi task learning.
Now I'm experimenting with incorporating the 3rd loss function into the same model with the first 2.
My challenge is that the 3rd loss function is only valid when the 1st classifier identifies a positive sample (it's a regression output that measures the identified sample). These labels are known at training time.
- Standardizing the labels and impute a 0 value for the 3rd regression for all negative samples. That approach biases towards 0 in some cases, so I threw it out.
- Alternate training between all samples on the 1st and 2nd loss, and only positive samples using the sum of all 3 loss functions. Mixed results here, but the 3rd loss function seems to perform worse than it does in its own separate model.
I'm considering how I might re-structure the loss function (currently L2 loss) so that it treats a 0-value as irrelevant (e.g. no gradient is propagated when the label is 0).