Missing outputs in multiple-output neural net

Question

I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was to use a neural network (e.g. adding multiple outputs to a pre-trained covnet) and I would try keras first (having used it before). I assume it would be useful to try to do this all in a single covnet (especially based on a TWiML talk on this paper, where the author suggested that in general one prediction target will help to improve features for another and given that e.g. how one would recognise gender will differ by breed etc.).

My question arises, because for some training/validation/test images some (but not all) of the multiple prediction targets may be missing. Sometimes we will just know the breed, for some images we will not know the gender and the exact age may oftne be unavailable (but only that the animal is adult versus younger than about 4 months, which a human can easily label by just looking at the animal).

If I had to guess, I would expect the missingess to be missing at random (but likely not missing completely at random). For example, for some breeds and ages it is harder to tell the gender by just looking at the animal (perhaps the neural network will manage to do it). So, someone who just takes a photo of an animal may not know the gender and humans are no good at labelling gender from photos. Another example: age might be unavailable, because someone just took a photo of an animal they do not have detailed information on or because someone bought an animal secondhand and never found out.

Is there some standard way of handling this? My intuition is that even records with one or two missing prediction target still should provide useful information for the other ones. Thus, listwise deletion of all records with any missing target information would be extremely inefficient (and possibly even otherwise problematic?!).

Answer 1

Since we are talking about multiple different types of targets (classes versus numerical for example) we already need a composite loss function. I will consider how to balance the different composite parts of the loss function outside of the scope of this answer but if you look into multi-task learning there are solutions to this. What you could do (both in training and evaluation) is to mask the parts of the loss function that are unknown.

You can do this fairly easily (but hacky) by passing weights as input during training or prediction for each output in your sample whether it was available or not as 1 or 0 and multiply that component with the weight. For analysis of the training and prediction you need to make sure to take the mean of only the available labels because otherwise your loss values are too optimistic.

I do think this approach is very valuable in a lot of cases, this can also help with semi-supervised learning where you do have labels available for a similar task.