I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the
auc() of the
precision_recall_curve(), I get a different macro-metric than when I use the builtin
average_precision_score() function. As per the docs, this makes sense, due to
auc() using a trapezoidal approximation, and thus average precision being more accurate. However, when I compute the micro-metrics, the values are the same. I'm trying to understand this.
Here's an example:
yhat_raw = np.array([[0.511, 0.62, 0.3],[0.11, 0.76, 0.945],[0.002, 0.671, 0.452],[0.9102, 0.783, 0.2]]) y = np.array([[0, 1, 1],[0, 0, 1],[0, 0, 0],[0, 1, 1]]) yhatmic = yhat_raw.ravel() ymic = y.ravel() prec["micro"], rec["micro"], _ = precision_recall_curve(ymic, yhatmic, pos_label=1)
average_precision_score(ymic, yhatmic) returns
Upon actually deploying the model, these metrics are coming to the same thing. I think this makes sense, considering that at a larger scale (e.g. along a dimension of thousands of datapoints rather than
len(rec['micro']) = 11 datapoints in this example), the trapezoidal approximation is much closer to the true value.
My question is, why do the metrics then differ in the
macro case? Is it fair to say that in a real-world deployment, the micro-evaluation has enough datapoints for the trapezoidal approximation to converge to the true value, but the macro- does not (computed across ~3000 test examples, with a label space of ~2000 labels)?