How to create ROC - AUC curves for multi class text classification problem in Python

Question

I am working on a multiclass text classification problem and trying to plot ROC Curve but no success so far. Tried many solutions available but didn't work. Kindly please someone help me out with the following piece of code to plot the ROC curve. There are actually five different classes for which I am performing text classification.

categories = ['Philonthropists', 'Politcians', 'Showbiz', 'Sportsmen', 'Writers']
train = dt.load_files(r'C:\Users\...\Learning\Train', categories=categories, encoding='ISO-8859-1')
test = dt.load_files(r'C:\Users\...\Learning\Test', categories=categories, encoding='ISO-8859-1')
count_vector = CountVectorizer()
x_trian_tf = count_vector.fit_transform(train.data)
tfidf_transformer = TfidfTransformer()
x_train_tfidf = tfidf_transformer.fit_transform(x_trian_tf)
learn = MultinomialNB().fit(x_train_tfidf, train.target)
x_test_tf = count_vector.transform(test.data)
x_test_tfidf = tfidf_transformer.transform(x_test_tf)
prediction = learn.predict(x_test_tfidf)
print("Accuracy is of Multinomial Naive Bayes Classifier", accuracy_score(test.target, prediction) * 100)

Answer 1

First check out the binary classification example in the scikit-learn documentation. It's as easy as that:

from sklearn.metrics import roc_curve
from sklearn.metrics import RocCurveDisplay
y_score = clf.decision_function(X_test)

fpr, tpr, _ = roc_curve(y_test, y_score, pos_label=clf.classes_[1])
roc_display = RocCurveDisplay(fpr=fpr, tpr=tpr).plot()

In the case of multi-class classification this is not so simple. If you have 3 classes you could do ROC-AUC-curve in 3D. Have a look at the resources here.

What you can do and what is much simpler is to make 4 one-vs-all curves. You basically have a binary setting for each class.

In your case:

import matplotlib.pyplot as plt
# all the same up until now
prediction = learn.predict(x_test_tfidf)
proba = learn.predict_proba(x_test_tfidf)
print("Accuracy is of Multinomial Naive Bayes Classifier", accuracy_score(test.target, prediction) * 100)

for i in range(len(categories)):
    y_test_bin = np.int32(test.target == i)
    y_score = proba[:,i]
    fpr, tpr, _ = roc_curve(y_test_bin, y_score, pos_label=0)
    plt.subplot(2,2,i)
    roc_display = RocCurveDisplay(fpr=fpr, tpr=tpr).plot()