I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was to use a neural network (e.g. adding multiple outputs to a pre-trained covnet) and I would try keras first (having used it before). I assume it would be useful to try to do this all in a single covnet (especially based on a TWiML talk on this paper, where the author suggested that in general one prediction target will help to improve features for another and given that e.g. how one would recognise gender will differ by breed etc.).
My question arises, because for some training/validation/test images some (but not all) of the multiple prediction targets may be missing. Sometimes we will just know the breed, for some images we will not know the gender and the exact age may oftne be unavailable (but only that the animal is adult versus younger than about 4 months, which a human can easily label by just looking at the animal).
If I had to guess, I would expect the missingess to be missing at random (but likely not missing completely at random). For example, for some breeds and ages it is harder to tell the gender by just looking at the animal (perhaps the neural network will manage to do it). So, someone who just takes a photo of an animal may not know the gender and humans are no good at labelling gender from photos. Another example: age might be unavailable, because someone just took a photo of an animal they do not have detailed information on or because someone bought an animal secondhand and never found out.
Is there some standard way of handling this? My intuition is that even records with one or two missing prediction target still should provide useful information for the other ones. Thus, listwise deletion of all records with any missing target information would be extremely inefficient (and possibly even otherwise problematic?!).