3
$\begingroup$

I am working on a multiclass text classification problem and trying to plot ROC Curve but no success so far. Tried many solutions available but didn't work. Kindly please someone help me out with the following piece of code to plot the ROC curve. There are actually five different classes for which I am performing text classification.

categories = ['Philonthropists', 'Politcians', 'Showbiz', 'Sportsmen', 'Writers']
train = dt.load_files(r'C:\Users\...\Learning\Train', categories=categories, encoding='ISO-8859-1')
test = dt.load_files(r'C:\Users\...\Learning\Test', categories=categories, encoding='ISO-8859-1')
count_vector = CountVectorizer()
x_trian_tf = count_vector.fit_transform(train.data)
tfidf_transformer = TfidfTransformer()
x_train_tfidf = tfidf_transformer.fit_transform(x_trian_tf)
learn = MultinomialNB().fit(x_train_tfidf, train.target)
x_test_tf = count_vector.transform(test.data)
x_test_tfidf = tfidf_transformer.transform(x_test_tf)
prediction = learn.predict(x_test_tfidf)
print("Accuracy is of Multinomial Naive Bayes Classifier", accuracy_score(test.target, prediction) * 100)
$\endgroup$
2
$\begingroup$

First check out the binary classification example in the scikit-learn documentation. It's as easy as that:

from sklearn.metrics import roc_curve
from sklearn.metrics import RocCurveDisplay
y_score = clf.decision_function(X_test)

fpr, tpr, _ = roc_curve(y_test, y_score, pos_label=clf.classes_[1])
roc_display = RocCurveDisplay(fpr=fpr, tpr=tpr).plot()

In the case of multi-class classification this is not so simple. If you have 3 classes you could do ROC-AUC-curve in 3D. Have a look at the resources here.

What you can do and what is much simpler is to make 4 one-vs-all curves. You basically have a binary setting for each class.

In your case:

import matplotlib.pyplot as plt
# all the same up until now
prediction = learn.predict(x_test_tfidf)
proba = learn.predict_proba(x_test_tfidf)
print("Accuracy is of Multinomial Naive Bayes Classifier", accuracy_score(test.target, prediction) * 100)

for i in range(len(categories)):
    y_test_bin = np.int32(test.target == i)
    y_score = proba[:,i]
    fpr, tpr, _ = roc_curve(y_test_bin, y_score, pos_label=0)
    plt.subplot(2,2,i)
    roc_display = RocCurveDisplay(fpr=fpr, tpr=tpr).plot()
$\endgroup$
  • $\begingroup$ Thank you so much for this answer. I know that ROC is basically for binary classification but I already tried many solutions for the multi-class ROC curve but none of them worked for me. Could you please amend my code according to what you have suggested above? Bundle of thanks in advance. $\endgroup$ – Muneeb Jul 4 '20 at 7:55
  • $\begingroup$ Updated my answer. Not sure how test.target looks like, I assume it's a 1D numpy array of class labels. $\endgroup$ – Tinu Jul 4 '20 at 8:36
  • $\begingroup$ I'm getting this eror while executing code "C:\Users\mr.geek\AppData\Local\Programs\Python\Python37-32\lib\site-packages\matplotlib\axes_subplots.py", line 66, in init f"num must be 1 <= num <= {rows*cols}, not {num}") ValueError: num must be 1 <= num <= 4, not 0 $\endgroup$ – Muneeb Jul 4 '20 at 13:00
  • $\begingroup$ My bad, use plt.subplot(2,2,i+1). $\endgroup$ – Tinu Jul 4 '20 at 14:41
  • $\begingroup$ Now after changing plt.subplot(2, 2, i+1) now I am getting this error. ValueError: num must be 1 <= num <= 4, not 5 $\endgroup$ – Muneeb Jul 4 '20 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.