mAP scores on tensorboard (Tensorflow Object Detection API) are all 0 even though the loss value is low

Question

I trained a faster-rcnn model on the tensorflow object detection API on a custom dataset. I found that the loss is ~2 after 3.5k steps. However, when I ran eval.py, the mAP scores are all almost 0 as shown below.

I do not understand why this is the case. However, when I look at the images at 3.5k steps, the model has captured some of the boxes as shown below

Can someone please explain why the mAP scores are close to zero, even though the model has learned to output quite a few boxes?

Answer 1

I recommend several checks to make sure you get reasonable mAP@IoU scores for object detection API:

1. Try varying the Intersection over Union (IoU) threshold, e.g 0.2-0.5 and see if you get an increase in average precision. You would have to modify matching_iou_threshold parameter in object_detection/utils/object_detection_evaluation.py
2. Try different evaluator classes (the default one is EVAL_DEFAULT_METRIC = 'pascal_voc_detection_metrics'). If you are training on Open Image Dataset it makes sense to use open_images_V2_detection_metrics

3. Check your eval config file and increase the number of examples used in the evaluation set, e.g.

   eval_config: { num_examples: 20000 num_visualizations: 16 min_score_threshold: 0.2 # Note: The below line limits the evaluation process to 10 evaluations. # Remove the below line to evaluate indefinitely. max_evals: 1 }

4. Train the object detector for more iterations
5. Check current mAP against reported metrics (e.g. COCO mAP@IoU=0.5):