Object Detection: Setting threshold values as trainable parameters?

Question

I am building my first object detection model (Mobilenet SSD, to detect animals in images) and happy with the current test results.

When I tested it using images without bounding boxes, I noticed some of the images get multiple object detections around parts of the object of interest (example, two bounding boxes around features of a cat, or one high confidence cat bounding box and a slightly lower confidence dog bounding box).

I have manually set threshold values for each class (example, only cats that have a confidence of over 90% are detected) and added some additional logic (similar to non-Maximum Suppression (NMS)), but I am concerned that this is not the best approach as the model's final performance is not necessary aligned with the test results (since there is this additional threshold and logic checks that are not incorporated during training).

Previously when I worked with just image classification, there was no additional logic that I needed to add because of the lack of bounding boxes.

Do I need to set bounding box thresholds as trainable parameters for the network? If so, how? Am I following the right approach?

Answer 1

I'm going through the same problems and it seems that thresholds (confidence or IoU for NMS) are just hyperparameters that you need to set. You need to decide whether you want a higher precision or higher recall.

This is a good comment by another user on SO that explains this stuff in much more detail: https://stackoverflow.com/a/76086662/16629499

Based on this answer, and some other discussions like this, it seems that you first need to:

1. Decide if you want a higher precision or recall
2. Apply the object detector on your validation data using different thresholds (The standard thresholds used for object detection are from 0.5:0.95 in intervals of 0.05)
3. Sort the predictions in the decreasing order of their scores and plot a precision-recall point for each subset
4. This curve will help you decide the optimal threshold/confidence values that you need to select based on your desired precision-recall tradeoff.