I have a network whose output is a large vector ( ~ 2000 let's say ). The ground truth output is very sparse and binary- that is out of the 2000 most of the values will be 0 and just a handful will be 1. Just to clarify, each sample will have 1's at different locations with some uneven likelihood of there being a 1 at any given neuron.

I am pretty sure that if I run this network it will quickly find the trivial solutions of outputting all 0 and just stay there.

How might you suggest to nudge the network to avoid this trivial solution?

I thought of somehow weighting the samples in such a way that each sample will give a high weight to the neurons that are equal to 1. This seems very hard to implement efficiently. How can I easily implement this in tensorflow/keras efficiently?

Another Idea is to set all the 0's to random centered at .5 and slowly decrease them towards zero as the network trains.

Other ideas?


Some interesting suggestions I got from others:

  1. split the network so that each subnetwork is in charge of a subset of the classes.
  2. use the Dice loss or jaccard loss which are pretty good for this situation.
  3. pre-train with the less prevalent classes only and add classes over epochs. This needs to be done very carefully.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.