Logistic regression with unbalanced data, scoring based only on rare class

Question

I have a dataset off app. 600.000 data points in which 0.2% (1.200 samples) is labelled as signifying a rare event. I want to use logistic regression to help me predict this rare event, but even when I apply weighting, the classification accuracy is poor.

I know that I can rebalance the dataset, but the problem with that is that lots of other weird stuff is going on that signifies various events but which is not indicative of the exact type of event I’m trying to predict. Therefore, rebalancing would itself be a large and complicated task in trying to get representative weirdness into the cropped dataset.

I have found that I get great predictive performance by using a home-made logistic regression classifier that scores based on the error measure below, and then using scipy.optimimize.minimize with method='Powell' to tune the parameters to minimize this measure:

e = 1 - tp/(tp+fp+fn)

for the rare class only, ignoring true negatives.

This lets me include the full dataset (and ignore weights and rebalancing considerations), and it makes it clear in the scoring when the classifier starts erroneously including other types of events that shouldn’t be caught by this classifier (i.e. increasing number of false positives).

The problem is that my home-grown classifier/trainer is naturally much slower to train than the sklearn version, and I’d like to build on mature optimized packages instead. So my question is: Is there a Python implementation of a logistic regression classifier that lets me perform fitting based only on the rare class as described above? It seems that setting the rare class weight arbitrarily high does not make it perform like my home-grown method that explicitly ignores the common class.

Answer 1

Firstly, when you have an imbalanced dataset accuracy is not a good metric to be using (see https://en.wikipedia.org/wiki/Precision_and_recall#Imbalanced_data). You should consider what the ultimate use-case of this model is and what metric is properly capturing the performance of the model considering that use case. For example, when classifying the presence of cancer, false negatives are much more undesirable than false positives so you would want to ensure you are using a metric that captures that appropriately.

In sklearn there is a class_weight parameter of the LogisticRegression model which allows you to essentially weigh misclassifications of different classes differently. Setting this to 'balanced' will automatically adjust this weight to be inversely proportional to the amount of samples of that class in your data which might be beneficial. You may want to adjust this in a custom manner also.

Changing the metric you are evaluating on doesn't change the actual training of the model, so I am guessing that your custom implementation of logistic regression should not function significantly differently to the sklearn version in terms of performance (if it does their may be other issues), it seems you are just using a different metric. There are also a number of other metrics besides accuracy that you can use in sklearn (https://scikit-learn.org/stable/modules/model_evaluation.html#model-evaluation) perhaps consider balanced-accuracy to begin with. It is also not to hard to apply your own custom metric to the results from the sklearn logistic regression model.

Tools such as the classification_report (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) and/or confusion matrix (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) can also be enlightening when dealing with imbalanced data.