1
$\begingroup$

Is it me or has there been an increasingly large amount of object detection papers describing models that are performing worse than chance. Here is an example (an extract so not to name names):-

enter image description here

AP represents the average precision. No mention of recall.

The paper goes on to say that YOLOv3-Lite reaches state of the art performance in detecting the specified object. This table would suggest to me that the models are worse just flipping a coin. What am I missing here?

$\endgroup$
1
  • $\begingroup$ Hi, what exactly is the detection task in this example? $\endgroup$
    – Jonathan
    Commented Feb 17, 2020 at 23:44

2 Answers 2

3
$\begingroup$

Not all probabilities are 50/50.

I'm assuming that the paper you are looking at is eYOLOv3-Lite: A Lightweight Crack Detection Network.

Under the evaluation metrics section, I see

For each image, the intersection over union ($I_{oU}$) between the bounding box of the detected crack and ground truth can be calculated as: $I_{oU}$ = $\frac{A_o}{A_u}$ , where $I_{oU}$ is the intersection over union, $A_o$ is the area of overlap, and $A_u$ is the area of union. When the $I_{oU}$ of the predicted bounding box and ground truth is greater than a certain threshold value (e.g., 0.5), it is considered to be a true positive; otherwise, it is a false positive.

It looks like the problem is to assign a bounding box to each image such that the bounding box contains the crack (or the ground truth box associated with the crack). It's not an easy probability to calculate exactly, but it's quite clear that assigning a bounding box completely at random will yield a success much lower than 50%.

$\endgroup$
1
  • $\begingroup$ Thanks for this reply, this makes a lot more sense. I had assumed that the bounding box could possibly consist of the entire image and thought then probability would be 50:50 but in realty would result in a very low intersection of union value. $\endgroup$ Commented Feb 20, 2020 at 5:19
0
$\begingroup$

This table would suggest to me that the models are worse just flipping a coin

Why do you say that? Flipping a coin and expecting 50% Heads is random, however, this is true only when the probability of outcome is of one of two possibilities.

When the possible outcomes are more (than 2 for example), the random chance goes down. with the fraction being 1/N if there are N possible outcomes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.