What does the notation mAP@[.5:.95] mean?

Question

For detection, a common way to determine if one object proposal was right is Intersection over Union (IoU, IU). This takes the set $A$ of proposed object pixels and the set of true object pixels $B$ and calculates:

$$IoU(A, B) = \frac{A \cap B}{A \cup B}$$

Commonly, IoU > 0.5 means that it was a hit, otherwise it was a fail. For each class, one can calculate the

• True Positive ($TP(c)$): a proposal was made for class $c$ and there actually was an object of class $c$
• False Positive ($FP(c)$): a proposal was made for class $c$, but there is no object of class $c$
• Average Precision for class $c$: $\frac{\#TP(c)}{\#TP(c) + \#FP(c)}$

The mAP (mean average precision) = $\frac{1}{|classes|}\sum_{c \in classes} \frac{\#TP(c)}{\#TP(c) + \#FP(c)}$

If one wants better proposals, one does increase the IoU from 0.5 to a higher value (up to 1.0 which would be perfect). One can denote this with mAP@p, where $p \in (0, 1)$ is the IoU.

But what does mAP@[.5:.95] (as found in this paper) mean?

Answer 1

mAP@[.5:.95](someone denoted mAP@[.5,.95]) means average mAP over different IoU thresholds, from 0.5 to 0.95, step 0.05 (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

There is an associated MS COCO challenge with a new evaluation metric, that averages mAP over different IoU thresholds, from 0.5 to 0.95 (written as “0.5:0.95”). [Ref]

We evaluate the mAP averaged for IoU ∈ [0.5 : 0.05 : 0.95] (COCO’s standard metric, simply denoted as mAP@[.5, .95]) and [email protected] (PASCAL VOC’s metric). [Ref]

To evaluate our final detections, we use the official COCO API [20], which measures mAP averaged over IOU thresholds in [0.5 : 0.05 : 0.95], amongst other metrics. [Ref]

BTW, the source code of coco shows exactly what mAP@[.5:.95] is doing:

self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)

References

• cocoapi

• Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

• Speed/accuracy trade-offs for modern convolutional object detectors

Answer 2

You already have the answer from Icyblade. However, I want to point out that your Average Precision formula is wrong. The formula $\frac{\#TP(c)}{\#TP(c) + \#FP(c)}$ is the definition of precision, not Average Precision. For object detection, AP is defined in here. Briefly, it summarises the precision/recall curve hence not only precision but also recall is taken into account (hence the False Negative will be penalised too).

Answer 3

AP is averaged over all categories. Traditionally, this is called "mean average precision" (mAP). We make no distinction between AP and mAP (and likewise AR and mAR) and assume the difference is clear from context.

http://cocodataset.org/#detections-eval