3
$\begingroup$

I'm trying to analyze the performance of a binary classifier on the test set on different ranges of the predictions. the classifier has a .97 ROC AUC on the test. Then I binarize the test set predictions into bins to check the ROC AUC on every bucket but in the bins, it has very low performance.

Reproducible example:

import numpy as np
import pandas as pd

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split


n = 50
X, y = make_classification(n_samples  = 10000,n_features = n, n_informative=27,n_classes = 2, random_state = 42)

X = pd.DataFrame(X, columns = [f"x_{i}" for i in range(n)])
y = pd.DataFrame(y, columns = ["target"])

frame = pd.merge(X,y, left_index = True, right_index = True)
frame.loc[:,"split"] = np.random.choice(a = ["train","test"], p = [.7,.3], size = frame.shape[0])
train_df = frame.query("split == 'train'").drop("split", axis = 1)
test_df = frame.query("split == 'test'").drop("split", axis = 1)



clf = RandomForestClassifier(random_state = 42).fit(train_df.drop("target", axis = 1),train_df["target"])

preds = clf.predict_proba(test_df.drop("target", axis = 1))[:,1]

roc_ = roc_auc_score(y_true = test_df["target"], y_score = preds)

print(f"ROC AUC: {roc_}")

test_df.loc[:,"prediction"] = clf.predict_proba(test_df.drop("target", axis = 1))[:,1]

test_df.loc[:,"band"] = pd.qcut(q = 10, x = test_df.prediction, duplicates = "drop")

def get_auc_(y_true, y_score):

  try:
    return roc_auc_score(y_true = y_true, y_score = y_score)
  except:
    return np.NaN

test_df.groupby("band").apply(lambda x: get_auc_(y_true = x["target"], y_score = x["prediction"]))


band
(0.009000000000000001, 0.16]         NaN
(0.16, 0.23]                    0.051780
(0.23, 0.3]                     0.592401
(0.3, 0.39]                     0.633804
(0.39, 0.5]                     0.626548
(0.5, 0.61]                     0.629141
(0.61, 0.7]                     0.633596
(0.7, 0.77]                     0.702138
(0.77, 0.84]                    0.477372
(0.84, 0.98]                    0.480072
dtype: float64

My question is what explains this low score in the different bins?

$\endgroup$
6
  • $\begingroup$ Interesting question, but why do you bin the data? Also, what do you bin, the probability outputs of your model? // Might you be interested in calibration of the probability outputs? $\endgroup$
    – Dave
    Jun 7 at 16:20
  • $\begingroup$ Thanks for your comment, I'm interested in how well the model performs on each bucket of the predictions, that is to say if the model performs better with the riskiest or least risky according to ROC AUC. Rather than calibration, I'm interested on the performance itself. $\endgroup$ Jun 7 at 16:28
  • $\begingroup$ Calibration is part of performance, so I do not follow your comment. $\endgroup$
    – Dave
    Jun 7 at 16:30
  • $\begingroup$ I mean, I'm by now only interested in performance in terms of ROC AUC on each bin, not brier_score (more related to the calibration property of a classifier) $\endgroup$ Jun 7 at 16:33
  • $\begingroup$ Why are you interested in the AUC in each bin? This seems like an XY problem where you have issue X that you think you can solve with method Y; then when you encounter issues with Y, you ask about Y instead of X (which I suspect is calibration, but maybe it isn't). $\endgroup$
    – Dave
    Jun 7 at 17:01

2 Answers 2

4
$\begingroup$

My two cents:

  • If the goal is to predict the different bins corresponding to the probability of predicting the positive class, then this seems a strange design: why use binary classification if the main outcome of interest is not the binary class? This might be better framed as a regression problem where the goal is to predict some score instead (multiclass classification could be considered but it's not great for an ordinal target).
  • I think it's wrong to study the bands of predicted probability with ROC. ROC makes sense for studying the whole range of probabilities on a dataset, not for an interval. This is because ROC is by definition about measuring how good the classifier is at placing the positive instances at one end of the range and the negative ones at the other (actually the AUC represents exactly this). Additionally it's very likely that some bands have very few instances,likely causing serious approximations in the resulting AUC score. So imho these scores are hardly interpretable, possibly even meaningless.
  • A simple way to observe what happens with these bins would be to plot them as a histogram, with colours showing the proportion/number of positive/negative instances.
$\endgroup$
4
  • $\begingroup$ I really appreciate your answer. My comments: The idea of converting the output probabilities into bins is because thresholds are assigned to every customer, so imagine you are giving a product to those customers with a risk less than .3, they happen to be the ones on the band 0 to 3 (etc) I have seen this approach for example on FICO credit score. The bands have ~10% of the sample each since they are constructed as deciles, what I noticed is that there are bins for which imbalances is almost 100:1. Can this be affecting? $\endgroup$ Jun 7 at 21:07
  • $\begingroup$ @user_2340102 I think that a credit score would usually be predicted as the main target of a regression task. Imho it's a bit risky to use the predicted probabilities for a binary classifier, because these can have different distributions depending on the algorithm. To also me it makes more sense to study the bands on the target itself. $\endgroup$
    – Erwan
    Jun 9 at 17:18
  • $\begingroup$ The question is not focused on discussing the validation of the analysis but on the metrics obtained at bin levels vs the overall. $\endgroup$ Jun 9 at 20:15
  • $\begingroup$ @user_2340102 I understand, but it's related: you use AUC because the main task is designed as binary classification. However I think that AUC is not an appropriate metric. Even if we keep the binary classification design, simply counting the proportion of positive instances would be more meaningful for describing these deciles ranges. $\endgroup$
    – Erwan
    Jun 9 at 21:19
1
+100
$\begingroup$

One interpretation of the AUROC is "the probability that a randomly selected positive instance is given a higher probability by the model than a randomly selected negative instance." With that in mind, it's clear that a good model will tend to have a much better AUROC over all instances than the AUROC for each bin: the former includes all the inter-bin pairs of instances, and the larger differences in their probabilities indicates the model's relative certainty, so those inter-bin pairs are much more likely to be put in the correct order.

Indeed, you could imagine shuffling the scores of the instances within each bin; as long as the model has done a decent job in creating the bins, the overall AUROC will still be reasonably good, but the AUROC inside these now-shuffled bins will all be approximately 0.5.

All that said, it of course is preferable that the AUROC inside the bins is useful. If you intend to sell the scores within the first few bins to one business, all they will care about is the rank-ordering within those bins. It's possible that a new model focusing only on those cases can do better (not being "distracted" by getting the other instances right), although then there's a big question about whether your first model is doing well enough at creating that population in the first place.

Here's a quick example notebook demonstrating the phenomenon with a synthetic dataset.

$\endgroup$
2
  • $\begingroup$ Do you have more documentation that can support your interesting ideas here? $\endgroup$ Jun 18 at 19:31
  • $\begingroup$ I'm not sure what documentation would help, but I've added a notebook verifying that the phenomenon occurs with a simple synthetic dataset. $\endgroup$
    – Ben Reiniger
    Jun 21 at 2:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.