Tweak machine learning algorithm in SciKit to optimize for recall

Question

I am given a dataset to detect fraud. Something similar like this: https://www.kaggle.com/code/imgremlin/4th-place-in-fraud-detection-from-zindi

The issue with SciKit machine learning algorithm is that it optimizes for accuracy, but I want lower its accuracy and optimize for recall so that frauds can be detected more accurately.

The issue with the dataset is that there are much more non-fraud cases, "0", than fraud cases , "1". ~ 10 to 1

Is there a way that I can tweak the SciKit algorithm so that it optimizes for recall?

Answer 1

It is always tough when you have extreme unbalance in you classes. If you are able to do so, consider oversampling the minority class or undersampling the dominant class to create better balance.

But beyond that you can use gridsearch to tune the model and target it on a 'recall' metric

sklearn.grid_search.GridSearchCV(clf, param_grid, scoring="recall")

You can read up on how here

Answer 2

Pursuing recall alone is not a well-defined decision rule: just classify everything as 1 and you'll get 100% recall. Actually, anything threshold-sensitive is not a very good optimization target, assuming you mean gridsearch-like routines.

If you have to work with recall, calculate sklearn.metrics.precision_recall_curve() and find a point with good enough recall and decent precision and use the respective threshold to compare your predict_proba() results to. Chances are, your model is already fine, you just need better decision making.

Answer 3

You can use the balanced accuracy metric for such cases. It is defined as the average of recall obtained on each class.

from sklearn.metrics import balanced_accuracy_score
y_true = [0, 1, 0, 0, 1, 0]
y_pred = [0, 1, 0, 0, 0, 1]
balanced_accuracy_score(y_true, y_pred)