I have an imbalanced dataset. Does it make sense to compute the roc-auc for the classifier I created in a holdout set?
Here's very artificial MWE:
from sklearn.datasets import load_iris from sklearn.linear_model import LogisticRegression X, y = load_iris(return_X_y=True) clf = LogisticRegression(solver="liblinear").fit(X, y) # Let's assume that X_test = X, y_test is just a vector of 1s. roc_auc_score(*150, clf.predict_proba(X_test), multi_class='ovr') ValueError: Number of classes in y_true not equal to the number of columns in 'y_score'