1
$\begingroup$

I created a classification model with three target classes and created a confusion matrix to measure the accuracy, here is the matrix code 

from sklearn.datasets import load_wine
data = load_wine()
x = pd.DataFrame(data=data.data, columns=data.feature_names )
y = pd.DataFrame(data=data.target, columns=['target'])
x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.2,shuffle=True,random_state = 42)
model = DecisionTreeClassifier()

model.fit(x_train, y_train)

y_pred = model.predict(x_test)

mat=confusion_matrix(y_test, y_pred, labels=[0,1,2])

The output of the above matrix is

array([[13,  1,  0],
       [ 0, 14,  0],
       [ 1,  0,  7]])

So, this is a balanced dataset and giving me an accuracy of 94% almost.

The problem is, when I tried to draw ROC AUC curve for class 0 using the below code, the AUC curve is the opposite and I am getting only 0.05 area under the curve.

fpr,tpr, thres =  roc_curve(y_test, y_pred, pos_label=0)
roc_auc = auc(fpr,tpr)

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()

I tried to calculate and draw ROC AUC manually and I got TPR as 0.9285 and FPR as 0.04545 and the graph is perfect on the sheet, could you please help me why the above code is giving the graph the other way. I verified the above code for the other two classes and the graph is good. Thanks in advance.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ The FPR and TPR must be swapped somewhere but I can't see where in the code you give. $\endgroup$
    – Erwan
    Commented Mar 20, 2020 at 0:57
  • $\begingroup$ How do you generate y_pred? Is it the same in the first and third code blocks? If so, that's probably the problem... $\endgroup$
    – Ben Reiniger ♦
    Commented Mar 20, 2020 at 14:18
  • $\begingroup$ @Erwan, I tried it while fitting roc_curve, but the function roc_curve() returns fpr first. I added my code to the question again, I appreciate your inputs. $\endgroup$
    – Shiva Borusu
    Commented Mar 20, 2020 at 18:19
  • $\begingroup$ @BenReiniger, I edited the question to make it more clear Ben, yes, the y_pred is same. $\endgroup$
    – Shiva Borusu
    Commented Mar 20, 2020 at 18:19

1 Answer 1

Reset to default
0
$\begingroup$

The ROC curve is built by taking different decision thresholds, and should be built using the predict_proba of your estimator. In particular, in your multiclass example, the ROC is using the values 0,1,2 as a rank-ordering! So there are four thresholds, the one between 0 and 1 being the most important here: there, you declare all of the samples the model predicts as being class 0 to be negative, and all others as positive, giving you the complementary point in the ROC-space.

Instead, you should have

fpr,tpr, thres =  roc_curve(y_test, model.predict_proba(x_test)[:,0], pos_label=0)
# or 
fpr,tpr, thres =  roc_curve(y_test==0, model.predict_proba(x_test)[:,0])
Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks Ben, I learned it now $\endgroup$
    – Shiva Borusu
    Commented Mar 21, 2020 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.