Area Under Curve with probability

Question

I'm coming across a metrics for model evaluation which I had never seen before and I don't know how to further research (since I don't know its proper name).

I'm using someone else's code, whose goal is to perform cross-validation to choose the best tree-based algorithm for a binary classification. It is probably worth saying that classes are highly skewed (93% / 7%). The metric which is used is the following: the classifier is trained and then the probability associated with each test element is computed.

probas = probas[:,list(clf.classes_).index(1)]

Then, these probabilities are ordered from the highest to the lowest and put in the x-axis. On the y-axis, being y the cumulative sum of entries associated with each probability. Then, they compute the area under the obtained curve, as in:

joint = zip(probas, truth)
joint = sorted(list(joint), key=lambda x:x[0], reverse=True)
probas = [x[0] for x in joint]
truth = [x[1] for x in joint]

# Calculate accumulated number of true labels at each probability point.
# Also calculate Area Under Curve (AUC) (higher is better model)
truth_cumulative = np.cumsum(truth) / np.sum(truth)
area = np.trapz(truth_cumulative, dx=1) / len(truth)

Can anyone give me an intuition of what this metric is about and a pointer to some resources to better understand it?

Thanks.

Answer 1

In your code the Area Under the Curve (AUC) is used to calculate the area under the Cumulative Distribution Function (CDF). Let's go through the code to see how this is done. However, in the last section, I do not agree with the two lines used to calculate the area. If all the labels have a ground truth label of 1, and we have a perfect classifier where the probability of being Class 1 is always 100%, we should have an AUC of 1. However, we will get 50%. I will show this later. You should note that the way this code is written will only work for a binary classifier, with labels 0 and 1.

We will first calculate the probability of each instance being in class 0 and class 1 by

probs = forest.predict_proba(X_train)

Then we will extract the second column, this contains the probability of each instance being in Class 1.

probas = probs[:,list(forest.classes_).index(1)]

Then we will join the probabilities of each instance with its ground truth label. Then we will sort the groupings based on their probability. The instances with a probability of 1 will be at the start of the list and those with a probability of 0 will be at the end of the list. This is done using

joint = zip(probas, truth)
joint = sorted(list(joint), key=lambda x:x[0], reverse=True)

Then we will separate the probabilities and the ground truth labels into separate lists using

probas = [x[0] for x in joint]
truth = [x[1] for x in joint]

The blue line is the probabilities and the ground truth labels is the orange line.

---

This is where the problems start. The next two lines do not include the probabilities but only the ground truth labels.

truth_cumulative = np.cumsum(truth)
truth_cumulative = truth_cumulative / np.sum(truth)

This means that if we had all class 1, with probability 1. We would get an AUC of 0.5. This does not make any sense.

For example, if I force it by doing this

probas[100:100000] = 1 truth[100:100000] = 1

There will only be a few 0 values left. Then after the ordering and cumulative sum we get the plot as follow, which is obviously wrong.

---

We need to do exactly what is written in the comment but not done through code. We will get the false positive rates for each false negative rate. Much like what is described below in the Receiver Operator Curve (ROC).

This can be done with

# Training Set
probas_train = forest.predict_proba(X_train)
fpr_t, tpr_t, thresholds_t = roc_curve(y_train, probas_train[:, 1])

# Testing Set
probas_ = forest.predict_proba(X_test)
fpr, tpr, thresholds = roc_curve(y_test, probas_[:, 1])
roc_auc = auc(fpr, tpr)

train, = plt.plot(fpr_t, tpr_t, label = 'Train')
test, = plt.plot(fpr, tpr, label = 'Test')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend()
plt.show()

print('Area Under the Curve (AUC): ', roc_auc)

Training Set
Area Under the Curve (AUC): 0.998140359403
Testing Set
Area Under the Curve (AUC): 0.937729144675

---

Receiver Operator Curve (ROC)

This was invented during WWII as a means to detect aircraft using radar. For example, if we have a sensor which requires some kind of threshold to detect planes. We can determine the true positive rate (TPR) and false positive rate (FPR) which results in our experiment. We will then plot these as the x-axis as the false positive rate, and the y-axis as the true positive rate.

$TPR = \frac{\text{True Positives}}{\text{Positives}}$

$FPR = \frac{\text{False Positives}}{\text{Negatives}}$

If the sensor is perfect we will always have $0$ false positives and always a $100\%$ true positives. This results in a curve which looks like the blue curve. If the sensor was completely random and garbage, then you would have a random guess which is the diagonal line.

A good classifier will be as close to the blue line as possible. Note, that a line below the diagonal is also good, you should just invert your decision. You have a consistent classifier but you are inverting the decision.

Intuitively, consider a sensor which reads a value and it is above a threshold $\theta$ we call it a detection. If we lower the threshold of detection, we should expect more detection, however if we end up with an equal amount of true positives and false negatives, then we are going along the diagonal line. This is a bad classifier.

Here we have 2 sensors, red and yellow, we want to decide which sensor is better. Just looking at the curves is not very effective because it's hard to see which one is definitely higher. So we can devise a metric named the Area Under the Curve (AUC). This is exactly what it means in the name. Then we an say the curve with the higher AUC is the better one.