Correctness of a ROC Curve

Question

I've built a Decision Tree Classifier to practice with the sklearn library. My first task was to shuffle the iris dataset and split it keeping only the last 10 elements for the test. Then, after the training part I predicted the class of these elements and printed other useful metrics to understand what I'm doing. My doubt comes after building the roc curve, since I find it is really different from the other I'v seen in some example. This is the code:

from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score
from sklearn.metrics import roc_curve, auc
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import cross_val_score
import matplotlib.pyplot as plt
from sklearn import tree 
import numpy as np
import graphviz 

iris = load_iris()

clf_ex1 = tree.DecisionTreeClassifier(criterion="entropy",random_state=300,min_samples_leaf=5,
                                      class_weight={0:1,1:10,2:10})

np.random.seed(0)

indices = np.random.permutation(len(iris.data))
indices_training=indices[:-10]
indices_test=indices[-10:]

iris_X_train = iris.data[indices_training]
iris_y_train = iris.target[indices_training]
iris_X_test  = iris.data[indices_test]
iris_y_test  = iris.target[indices_test]

clf_ex1 = clf_ex1.fit(iris_X_train, iris_y_train)

predicted_y_test = clf_ex1.predict(iris_X_test)

print("Predictions:")
print(predicted_y_test)
print("True classes:")
print(iris_y_test) 

# print some metrics results
acc_score = accuracy_score(iris_y_test, predicted_y_test)
print("--------")
print("Accuracy score: "+ str(acc_score))
print("--------")
f1=f1_score(iris_y_test, predicted_y_test, average='macro')
print("F1 score: "+str(f1))
print("--------")

scores = cross_val_score(clf_ex1, iris.data, iris.target, cv=5) 
print("Cross validation scores: "+str(scores))

# Confusion Matrix
print("--------")
print("Confusion matrix:")
print(confusion_matrix(iris_y_test, predicted_y_test))

# Building the ROC Curve 
y_test_prob = clf_ex1.predict_proba(iris_X_test)

# Calculating the roc curve for each class changing the pos_label value
fpr_cl0, tpr_cl0, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 0)
roc_auc_cl0 = auc(fpr_cl0, tpr_cl0)
fpr_cl1, tpr_cl1, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 1)
roc_auc_cl1 = auc(fpr_cl1, tpr_cl1)
fpr_cl2, tpr_cl2, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 2)
roc_auc_cl2 = auc(fpr_cl2, tpr_cl2)

# Building the Plot for each class
plt.figure()
lw = 2
plt.plot(fpr_cl0, tpr_cl0, color='darkorange',
         lw=lw, label='ROC curve class 0 (area = %0.2f)' % roc_auc_cl0)
plt.plot(fpr_cl1, tpr_cl1, color='cornflowerblue',
         lw=lw, label='ROC curve class 1 (area = %0.2f)' % roc_auc_cl1)
plt.plot(fpr_cl2, tpr_cl2, color='aqua',
         lw=lw, label='ROC curve class 2 (area = %0.2f)' % roc_auc_cl2)
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic example')
plt.legend(loc="lower right")
plt.show()

And these are the results:

1. Are they consistent with the predictions?
2. If I change the weights of the classes in my DecisionTreeClassifier, but I get the same predictions, is it normal that the final plot do not changes?

Answer 1

The first issue seems to be in the following block of code:

# Calculating the roc curve for each class changing the pos_label value
fpr_cl0, tpr_cl0, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 0)
roc_auc_cl0 = auc(fpr_cl0, tpr_cl0)
fpr_cl1, tpr_cl1, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 1)
roc_auc_cl1 = auc(fpr_cl1, tpr_cl1)
fpr_cl2, tpr_cl2, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 2)
roc_auc_cl2 = auc(fpr_cl2, tpr_cl2)

You are feeding y_test_prob[:,1] to each roc_auc calculator (when you should be feeding column 0, 1 and 2). In this case the predict_proba method outputs one column per class. Additionally you are feeding iris_y_test into this calculator which is numerically encoded with distinct values 0, 1 and 2. This function wants one column with the probabilities of each class on it's own. See a possible alteration:

import pandas as pd

# Calculating the roc curve for each class changing the pos_label value
fpr_cl0, tpr_cl0, _ = roc_curve(pd.get_dummies(iris_y_test).loc[:, 0], y_test_prob[:, 0])
roc_auc_cl0 = auc(fpr_cl0, tpr_cl0)
fpr_cl1, tpr_cl1, _ = roc_curve(pd.get_dummies(iris_y_test).loc[:, 1], y_test_prob[:, 1])
roc_auc_cl1 = auc(fpr_cl1, tpr_cl1)
fpr_cl2, tpr_cl2, _ = roc_curve(pd.get_dummies(iris_y_test).loc[:, 2], y_test_prob[:, 2])
roc_auc_cl2 = auc(fpr_cl2, tpr_cl2)

Secondly - looking at your output probabilities they are all either distinctly 0 or 1. Your decision tree seems to have "perfectly" classified each example into a class with 100% certainty.

The ROC AUC curve will only show interesting results if there are multiple possible thresholds to classify your samples into different classes. In your case - any threshold you pick between 0 and 1 exclusively will yield the exact same False Positive and True Positive rate. The ROC AUC curve helps to determine thresholds for classification - and in your case selecting thresholds is quite trivial.