I have a multiple object tracker that I apply on a specific object in an image series. The tracker has several parameters that can be adjusted which affects the performance of the tracking. I am testing the tracker on ground truth data and evaluate tracking performance by three different metrics:
- How close are the number of objects that the tracker outputs to the actual (ground truth) number of objects in the series. I want this to be as close the the actual value as possible. The value is an integer fixed to each series.
- How many false positive does the model predict. If the model fails to associate a detection to a group, it will consider it a false positive. However, there are no actual false positives in the test data, so this number should be as close to 0 as possible.
- The MOTA score. This score counts the number of mismatches the tracker made. The score ranges from -Inf to 1. 1 is a perfect score.
I have the idea that I would like to test out a large number of combinations of parameters in the tracking model (automatically, in Python) and then find the combination of parameters that give the overall best score of the three metrics above.
Let's say I want to give the three metrics the same weight. How would you do this?
I am thinking that I need to standardise the values in some way, for example to be between 0 and 1.
My idea so far is to calculate metric 1 as:
Where Nt is the number of objects returned by the tracker and Na is the actual number of objects
Metric 2 could be:
Where FPt is the number of false positives predicted by the tracker and P is the actual number of positives (sum of all detections).
I am not sure how to standardise this to a value between 0 and 1, where 0 is perfect.
Assuming that I have the last metric standardised, I would then sum these three scores and find the combination of metrics that give the sum closest to 0. Is this a viable approach to the problem?