4
$\begingroup$

I trained a faster-rcnn model on the tensorflow object detection API on a custom dataset. I found that the loss is ~2 after 3.5k steps. However, when I ran eval.py, the mAP scores are all almost 0 as shown below.

Category-wise mAP scores on Tensorboard

I do not understand why this is the case. However, when I look at the images at 3.5k steps, the model has captured some of the boxes as shown below

output of the Faster-RCNN model after 3.5k iterations

Can someone please explain why the mAP scores are close to zero, even though the model has learned to output quite a few boxes?

$\endgroup$
7
$\begingroup$

I recommend several checks to make sure you get reasonable mAP@IoU scores for object detection API:

  1. Try varying the Intersection over Union (IoU) threshold, e.g 0.2-0.5 and see if you get an increase in average precision. You would have to modify matching_iou_threshold parameter in object_detection/utils/object_detection_evaluation.py
  2. Try different evaluator classes (the default one is EVAL_DEFAULT_METRIC = 'pascal_voc_detection_metrics'). If you are training on Open Image Dataset it makes sense to use open_images_V2_detection_metrics
  3. Check your eval config file and increase the number of examples used in the evaluation set, e.g.

    eval_config: { num_examples: 20000 num_visualizations: 16 min_score_threshold: 0.2 # Note: The below line limits the evaluation process to 10 evaluations. # Remove the below line to evaluate indefinitely. max_evals: 1 }

  4. Train the object detector for more iterations
  5. Check current mAP against reported metrics (e.g. COCO mAP@IoU=0.5): COCO mAP at IoU=0.5
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.