1
$\begingroup$

The following figure is from the last page in YOLOv3 paper highlighting how mAP is unfair metric for evaluating Object Detectors: YOLOv3 last page The figure shows two hypothetical Object Detector results which the author say they give the same perfect mAP, while visually the first detector is clearly more accurate than the other.

According to my understanding, the two detectors do not give the same mAP. This is how I calculate it for each detector:

Detector 1, 'Dog' class AP table:
 ______________________________________ 
| Object  | True? | Precision | Recall |
|_________|_______|___________|________|
| Dog_99% | Yes   |     1     |    1   |  
|_________|_______|___________|________|
Hence, AP_dog = 1

Detector 1, 'Person' class AP table:
 ________________________________________
| Object    | True? | Precision | Recall |
|___________|_______|___________|________|
|Person_99% | Yes   |     1     |    1   |  
|___________|_______|___________|________|
Hence, AP_person = 1
And by continuing doing so for the other 7 classes in the dataset, mAP=1. 

Detector 2, 'Dog' class AP table:
 ______________________________________ 
| Object  | True? | Precision | Recall |
|_________|_______|___________|________|
| Dog_48% | Yes   |     1     |    1   |  
|_________|_______|___________|________|
Hence, AP_dog = 1

Detector 2, 'Bird' class AP table:
 _______________________________________
| Object   | True? | Precision | Recall |
|__________|_______|___________|________|
| Bird_90% | Yes   |     1     |    1   | 
| Bird_89% | No    |     0.5   |    1   | 
|__________|_______|___________|________|
Hence, AP_bird = 0.75
And by continuing doing so for the other 7 classes in the dataset, mAP is less than 1 because AP for at least one class is less than one (AP_bird).

Hence, according to my understanding mAP for the first detector is 1, and for the second detector is less than 1. What is the mistake I'm doing in the calculation? Or is there some assumption in the paper that I'm not considering?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.