Multi-task learning, finding a loss function that "ignores" certain samples

Question

I'm training a convolutional neural network with 1 input, and 3 outputs: a classification and 2 regression outputs.

I initially used 3 separate models, but I found that combining the 1st and 2nd loss functions into 1 model improves accuracy/rmse for both. I'm applying multi task learning.

Now I'm experimenting with incorporating the 3rd loss function into the same model with the first 2.

My challenge is that the 3rd loss function is only valid when the 1st classifier identifies a positive sample (it's a regression output that measures the identified sample). These labels are known at training time.

I attempted:

1. Standardizing the labels and impute a 0 value for the 3rd regression for all negative samples. That approach biases towards 0 in some cases, so I threw it out.
2. Alternate training between all samples on the 1st and 2nd loss, and only positive samples using the sum of all 3 loss functions. Mixed results here, but the 3rd loss function seems to perform worse than it does in its own separate model.

Question:

I'm considering how I might re-structure the loss function (currently L2 loss) so that it treats a 0-value as irrelevant (e.g. no gradient is propagated when the label is 0).

Any thoughts?

Answer 1

I seem to have a workable solution using this minor tweak to square loss:

$L=(y-\hat{y})^2 \cdot \left(\frac{y}{y + \epsilon}\right)$

Where $y$ is the label and $\hat{y}$ is the prediction, and $\epsilon=1e-6$ is to avoid a divide by zero error (it's ~ the smallest number within 32 bit precision)

In tensorflow:

epsilon = 1e-6
loss = tf.reduce_sum(
          tf.mul(tf.square(prediction - label),
                 tf.div(label, label + epsilon))
)

The second term is 0 when the label is 0 (e.g. loss is 0 and no gradient is added for that sample), and it'll round to 1, or near enough, when the label is non zero. As long as the range of labels does not cross 0 this should work, and that's a simple matter of scaling the labels if it's not the case by default.

Answer 2

A more simple answer than what I posted in 2017 is to multiply the loss function by a {1,0} mask. That will zero out the gradient when you multiply by 0 because 0dx = 0, and it won't change anything when multiplying by 1 because 1dx = dx. The {1,0} mask is a constant, treated the same as the labels that are input to the model.