I have a dataset off app. 600.000 data points in which 0.2% (1.200 samples) is labelled as signifying a rare event. I want to use logistic regression to help me predict this rare event, but even when I apply weighting, the classification accuracy is poor.
I know that I can rebalance the dataset, but the problem with that is that lots of other weird stuff is going on that signifies various events but which is not indicative of the exact type of event I’m trying to predict. Therefore, rebalancing would itself be a large and complicated task in trying to get representative weirdness into the cropped dataset.
I have found that I get great predictive performance by using a home-made logistic regression classifier that scores based on the error measure below, and then using scipy.optimimize.minimize with method='Powell' to tune the parameters to minimize this measure:
e = 1 - tp/(tp+fp+fn)
for the rare class only, ignoring true negatives.
This lets me include the full dataset (and ignore weights and rebalancing considerations), and it makes it clear in the scoring when the classifier starts erroneously including other types of events that shouldn’t be caught by this classifier (i.e. increasing number of false positives).
The problem is that my home-grown classifier/trainer is naturally much slower to train than the sklearn version, and I’d like to build on mature optimized packages instead. So my question is: Is there a Python implementation of a logistic regression classifier that lets me perform fitting based only on the rare class as described above? It seems that setting the rare class weight arbitrarily high does not make it perform like my home-grown method that explicitly ignores the common class.