I'm training a neural net on a "needle in haystack" problem - there is a huge amount of data, also with a huge class imbalance (~1 positive per 10k negatives). If I train the net on examples drawn so as to balance the classes, it works, but either overfits on the positives (since it sees the few positive examples repeated so often), or doesn't see most of the negatives. The result of not seeing most of the negatives is that there is a number of false positives that crop up, even on examples which were in the training set but which were never used in training. Presumably if the network was trained on those that would fix the problem.

The partial solution: every once in a while during training, run inference on all training data, and pick the hard examples (either false positive or false negative); use a balanced sample of just those for the next stage. This does work, but is quite slow, and it is unclear how many to pick, with what thresholds, etc.

The idea: I would like to train so that the log loss only works if too far from the target. In other words, true=0 and predicted=0.01 should not create a gradient at all; that way the vast majority of true negative examples would just run through a mostly-trained network but basically not do anything; the few false positives would generate a gradient, counterbalanced by the negatives.

Has this been done before? Does it work? Is there a better approach in the same situation?

  • $\begingroup$ What happens if you use square loss (Brier score)? $\endgroup$
    – Dave
    Jul 8 '20 at 10:43
  • 1
    $\begingroup$ You can try dealing with the class imbalance with class weights instead of sub-sampling. And when you sample, you don't have to balance the classes entirely - event 1:10 is better than 1:10K. $\endgroup$ Jul 8 '20 at 10:47

Have you looked into focal loss?

enter image description here

The idea seems to be similar to what you are describing - If predictions (~0.8) is close to GT Label of 1, it does not add to the loss value.

  • $\begingroup$ Yep, that's it exactly. I may have rediscovered the wheel with my idea :) I wonder if there is any paper that has done a systematic search of the different kinds of loss function scaling vs loss funciton truncation vs hand-picking examples as in hard-example mining. From the graph, I'd probably need one of the more extreme kinds of scaling like gamma=5 $\endgroup$
    – Alex I
    Jul 8 '20 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.