I'm training a neural net on a "needle in haystack" problem - there is a huge amount of data, also with a huge class imbalance (~1 positive per 10k negatives). If I train the net on examples drawn so as to balance the classes, it works, but either overfits on the positives (since it sees the few positive examples repeated so often), or doesn't see most of the negatives. The result of not seeing most of the negatives is that there is a number of false positives that crop up, even on examples which were in the training set but which were never used in training. Presumably if the network was trained on those that would fix the problem.
The partial solution: every once in a while during training, run inference on all training data, and pick the hard examples (either false positive or false negative); use a balanced sample of just those for the next stage. This does work, but is quite slow, and it is unclear how many to pick, with what thresholds, etc.
The idea: I would like to train so that the log loss only works if too far from the target. In other words, true=0 and predicted=0.01 should not create a gradient at all; that way the vast majority of true negative examples would just run through a mostly-trained network but basically not do anything; the few false positives would generate a gradient, counterbalanced by the negatives.
Has this been done before? Does it work? Is there a better approach in the same situation?
