0
$\begingroup$

I have a dataset off app. 600.000 data points in which 0.2% (1.200 samples) is labelled as signifying a rare event. I want to use logistic regression to help me predict this rare event, but even when I apply weighting, the classification accuracy is poor.

I know that I can rebalance the dataset, but the problem with that is that lots of other weird stuff is going on that signifies various events but which is not indicative of the exact type of event I’m trying to predict. Therefore, rebalancing would itself be a large and complicated task in trying to get representative weirdness into the cropped dataset.

I have found that I get great predictive performance by using a home-made logistic regression classifier that scores based on the error measure below, and then using scipy.optimimize.minimize with method='Powell' to tune the parameters to minimize this measure:

e = 1 - tp/(tp+fp+fn)

for the rare class only, ignoring true negatives.

This lets me include the full dataset (and ignore weights and rebalancing considerations), and it makes it clear in the scoring when the classifier starts erroneously including other types of events that shouldn’t be caught by this classifier (i.e. increasing number of false positives).

The problem is that my home-grown classifier/trainer is naturally much slower to train than the sklearn version, and I’d like to build on mature optimized packages instead. So my question is: Is there a Python implementation of a logistic regression classifier that lets me perform fitting based only on the rare class as described above? It seems that setting the rare class weight arbitrarily high does not make it perform like my home-grown method that explicitly ignores the common class.

$\endgroup$

1 Answer 1

1
$\begingroup$

Firstly, when you have an imbalanced dataset accuracy is not a good metric to be using (see https://en.wikipedia.org/wiki/Precision_and_recall#Imbalanced_data). You should consider what the ultimate use-case of this model is and what metric is properly capturing the performance of the model considering that use case. For example, when classifying the presence of cancer, false negatives are much more undesirable than false positives so you would want to ensure you are using a metric that captures that appropriately.

In sklearn there is a class_weight parameter of the LogisticRegression model which allows you to essentially weigh misclassifications of different classes differently. Setting this to 'balanced' will automatically adjust this weight to be inversely proportional to the amount of samples of that class in your data which might be beneficial. You may want to adjust this in a custom manner also.

Changing the metric you are evaluating on doesn't change the actual training of the model, so I am guessing that your custom implementation of logistic regression should not function significantly differently to the sklearn version in terms of performance (if it does their may be other issues), it seems you are just using a different metric. There are also a number of other metrics besides accuracy that you can use in sklearn (https://scikit-learn.org/stable/modules/model_evaluation.html#model-evaluation) perhaps consider balanced-accuracy to begin with. It is also not to hard to apply your own custom metric to the results from the sklearn logistic regression model.

Tools such as the classification_report (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) and/or confusion matrix (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) can also be enlightening when dealing with imbalanced data.

$\endgroup$
4
  • 1
    $\begingroup$ Sorry, I wasn't being completely clear in my original version of my problem statement. I've now rephrased it to (hopefully) show how my approach differs from the standard approach. Also, I've already tried changing the class weights, and that gave worse predictive performance than my current approach. $\endgroup$
    – Nick W
    Apr 22, 2021 at 5:09
  • $\begingroup$ What metric are you using to evaluate 'predictive performance'. Your custom metric is constructed in such a way that it will bias the model towards predicting negative, which will naturally achieve a high accuracy on the imbalanced dataset. So comparing the two model's predictive performance in terms of accuracy, or your custom error metric will naturally favor your custom optimization approach. But I don't think either of these metrics are good metrics to be using. What happens if you compare the results of the two models in terms of balanced accuracy? $\endgroup$
    – James
    Apr 22, 2021 at 5:57
  • $\begingroup$ To expand on that answer, tuning the class_weight parameter in logistic regression allows you to manage how much you care about false positives vs false negatives (Precision vs Recall). Weighing it heavily on the rare events (which we will consider a positive event) will favor recall. A very high recall will lead to poor accuracy, and will also perform badly in terms of your custom error metric. However, high recall will mean that you achieve less false negatives. Typically, in the sort of imbalanced dataset you are dealing with, we care more about false negatives than false positives. $\endgroup$
    – James
    Apr 22, 2021 at 6:25
  • 1
    $\begingroup$ Looking at my code again, it occurs to me that I'm also doing some pseudo time-series classification/scoring that makes my problem different from pure logistic regression. So your answer actually answers the question as posed, it's just me that didn't remember the differences clearly enough. Thanks for the answer, and sorry for the lack of clarity on my part. $\endgroup$
    – Nick W
    Apr 22, 2021 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.