0
$\begingroup$

Why do these below lines give different outputs while the input is the same? I need to report these results in paper, but I am unsure which is better and why.

y_pred_list=classifier.predict(X_test)
preds = classifier.predict_proba(X_test)
print(y_pred_list[0:10])
print(preds[:,1][0:10])

output:

[1. 1. 0. 0. 1. 1. 1. 1. 1. 1.]
[0.7348952  0.9941888  0.4657593  0.00950034 0.5141634  0.504719
 0.55126435 0.8347259  0.6122471  0.5445128 ]

Accuracy with y_pred_list

test_auc = roc_auc_score(y_score=y_pred_list, y_true=y_test)
print(test_auc) # => 0.7003153352913113

Accuracy with preds[:,1]

test_auc = roc_auc_score(y_score=preds[:,1], y_true=y_test)
print(test_auc) # => 0.7823557039988143

Screenshot for reference:

enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

There is a fundamental difference between .predict() and .predict_proba().

  • The former does argmax(logits), where the logits are raw class probabilities. Basically a predict returns the most probable class label.
  • The latter, predict_proba, instead, returns the probability that the positive class to occur. That is logits / logits.sum().

Now, the two are interpreted differently by roc_auc_score and so you get two different AUCs. Basically, when you call roc_auc_score(y_score=preds) the metrics is like thresholding preds in multiple ways to get the positive class: positive = preds > t. So at each threshold t, you get a different TPR (true positive rate) e FPS that, after integration, determine the AUC. In the first case, i.e. roc_auc_score(y_score=y_pred_list) the positive class is already determined meaning that y_pred_list is insentitive to the threshold t.

To conclude, you should use class scores like the probabilities to compute the AUC of either ROC or PR, i.e.: roc_auc_score(y_score=preds, ...). Instead, use y_pred_list to compute metrics, like accuracy, that already expect a class label and not a probability nor score.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.