3
$\begingroup$

I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the precision_recall_curve(), I get a different macro-metric than when I use the builtin average_precision_score() function. As per the docs, this makes sense, due to auc() using a trapezoidal approximation, and thus average precision being more accurate. However, when I compute the micro-metrics, the values are the same. I'm trying to understand this.

Here's an example:

yhat_raw = np.array([[0.511, 0.62, 0.3],[0.11, 0.76, 0.945],[0.002, 0.671, 0.452],[0.9102, 0.783, 0.2]])
y = np.array([[0, 1, 1],[0, 0, 1],[0, 0, 0],[0, 1, 1]])
yhatmic = yhat_raw.ravel()
ymic = y.ravel()
prec["micro"], rec["micro"], _ = precision_recall_curve(ymic, yhatmic, 
pos_label=1)

auc(rec["micro"], prec["micro"]) returns 0.58305555555555544. average_precision_score(ymic, yhatmic) returns 0.62222222222222223.

Upon actually deploying the model, these metrics are coming to the same thing. I think this makes sense, considering that at a larger scale (e.g. along a dimension of thousands of datapoints rather than len(rec['micro']) = 11 datapoints in this example), the trapezoidal approximation is much closer to the true value.

My question is, why do the metrics then differ in the macro case? Is it fair to say that in a real-world deployment, the micro-evaluation has enough datapoints for the trapezoidal approximation to converge to the true value, but the macro- does not (computed across ~3000 test examples, with a label space of ~2000 labels)?

$\endgroup$
0
$\begingroup$

The metrics differ in the macro case because macro in scikit-learn ignores class imbalances. micro in scikit-learn looks at global rates. The data has class imbalances, thus the macro argument will return different values compared to micro.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.