1
$\begingroup$

I have the mAP scores for an object detection model evaluated at different model checkpoints. I want to choose the model that has the maximum recall on the test dataset, meaning the model which missed the least number of objects. However, the model that achieves this doesn't have the highest mAP.

mAP is the most common metric for evaluating an object detection model. How do you interpret this metric?

For the best real world performance, is it better to:

  1. Choose a model checkpoint based on the least number of false negatives, even though the mAP isn't the highest?
  2. Choose a model checkpoint with the highest mAP, even if it means the number of false negatives is higher?
$\endgroup$

1 Answer 1

1
$\begingroup$

Choosing the model with the lowest loss on the validation set would be more appropriate. There are cases where the lowest loss model also exhibits the highest mAP, but that definitely is not the norm.

Regarding your question, if you are to choose the model based on accuracy-like metrics, consider what would be less harmful in your specific example. For instance, consider the case of cancer classification, predicting that a healthy patient has cancer (False Positive) might result in some frustration but it is not that big of a deal. On the other hand, predicting a cancer patient as of being healthy might cost his life.

$\endgroup$
2
  • $\begingroup$ Makes sense regarding which model to choose. If it is rare for the lowest loss model to have the highest mAP, then what does the mAP value tell you about how good the model is? For example, what does it mean for a model to have 60% mAP on the test set? Is there ever an instance where you choose a model based solely on its mAP? $\endgroup$ Apr 30, 2020 at 7:36
  • $\begingroup$ The model is trained to minimize the loss function you have defined on the training set. You can think of the loss function as a way of representing the accuracy in a differentiable way. Although they are not exactly the same. For instance, what would be the model's prediction of a healthy patient, let's say it outputs 0.85 | 0.15 healthy and cancer respectively what would have been the loss and the accuracy is this example? In terms of accuracy, a healthy patient is classified correctly as being healthy. On the contrary, the loss is not zero, since the model did not predict 1.0 | 0.0 . $\endgroup$ Apr 30, 2020 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.