# YOLOv1 algorithm - how to determine predictor responsibility

I am researching Yolo detector, and have read the original paper, but still have some confusion and a few questions regarding the assignment of bounding box predictors to ground truth at training time (in particular calculating the confidence).

Some resources like Andrew Ngs deep.ai here set the confidence of the corresponding label vector bounding box confidence to "1" if there is an object present at that grid cell. In contrast, here is an article from Oracles engineers that set the confidence to maximum Intersection-over-union value between some object and all bounding box predictors. I think the latter is also in line with what the paper is trying to tell us:

"YOLO predicts multiple bounding boxes per grid cell. At training time we only want one bounding box predictor to be responsible for each object. We assign one predictor to be “responsible” for predicting an object based on which prediction has the highest current IOU with the ground truth" (You Only Look Once: Unified, Real-Time Object Detection; Redmon, Divvala, Girshick, Farhadi)

1. Which of these methods should be used ?
2. I made a sketch to try to understand the mapping between ground truth and predictor bounding boxes. Please correct me if I'm wrong. The paper states we should assing a b.b. predictor to a ground truth object, but once we have done so, that bounding box predictor is "taken" and married to that object, so it can't be assigned to any other object in that grid cell? In my example with 3 ground truth bounding boxes and 2 predictors (B=2), we choose a predictor for object b1, which is b2_hat with highest IoU of 0.7, but b2 also has the highest Iou with b2_hat of 0.6, but b2_hat is already taken, so we take the next best thing, which is b1_hat. Now, all the bounding box predictors are assigned, and b3 gets no predictors, even if it has the highest IoU with all the boxes.

• I would advise to look at some implementations to see by yourself. There is several YoloV1 github repos. There are many potential matching methods actually. You could even use some optimal linear bipartite matching algorithm like the Hungarian algorithm (to find a matching that minimizes the sum of the costs). – Ismael EL ATIFI Nov 12 '20 at 20:53
• DETR (another object detection model whose paper is at arxiv.org/abs/2005.12872) uses the Hungarian algorithm to do the matching of GT boxes and predictors. Quote from the paper : "The usual solution is to design a loss based on the Hungarian algorithm [20], to find a bipartite matching between ground-truth and prediction." – Ismael EL ATIFI Nov 12 '20 at 21:08

Confidence score

Some resources like Andrew Ngs deep.ai here set the confidence of the corresponding label vector bounding box confidence to "1" if there is an object present at that grid cell.

This is because prof. Ng is refering to true probability of an object being in a specific cell ($$P(\text{object})$$ value), and not the confidence score ($$P(\text{Object}) \times\text{IOU}$$ value). Thereby, we have two possible situations: $$P(\text{object}) = \begin{cases} 1 \text{ if an object is present in the cell}\\ 0 \text{ otherwise} \end{cases}$$

However, this is not what YOLO is proposing to have as ground-truth value of confidence. Citing the paper:

[...] Formally, we define confidence as $$P(\text{Object}) \times\text{IOU}^{\text{truth}}_{\text{pred}}$$. If no object exists in that cell, the confidence scores should be zero [...]

Thereby, the value of confidence that the network is trying to predict (the confidence ground-truth label) is given by: $$P(\text{Object}) \times\text{IOU} = \begin{cases} \text{IOU if an object is present in the cell}\\ 0 \text{ otherwise} \end{cases}$$

Because of this, the other source mentioned in the question (Oracle's article) is the correct one.

Bounding boxes question

The paper states we should assing a b.b. predictor to a ground truth object, but once we have done so, that bounding box predictor is "taken" and married to that object, so it can't be assigned to any other object in that grid cell?

During training I don't see why this can be a problem. If the predicted bounding box of a certain grid cell, has the highest IOU for two or more ground-truth bounding boxes (whose center fall on this cell), we can interpret that the predicted bounding box is trying to predict both ground-truth boxes (it may happen for example if both ground truth boxes have similar sizes/ aspect ratios).

Concretely, what's said by the authors is the following:

YOLO predicts multiple bounding boxes per grid cell. At training time we only want one bounding box predictor to be responsible for each object. We assign one predictor to be “responsible” for predicting an object based on which prediction has the highest current IOU with the ground truth. This leads to specialization between the bounding box predictors. Each predictor gets better at predicting certain sizes, aspect ratios, or classes of object, improving overall recall.

The part in bold only implies that a ground-truth object is related only with one predicted bounding box $$\Rightarrow$$ a predicted bounding box can be related to more than one ground-truth object.

Edit:

Given what's said in the article, during training if a predicted bounding box has the highest intersection over union with two different objects, then the loss function will be related to both ground-truth bounding boxes.

For example, if we have a look at the part of the loss function related to the prediction of the center of the bounding box $$(\hat{x}_i, \hat{y}_i)$$:

$$\lambda_{coord}\sum_{i=0}^{S^2}\sum_{j=0}^B1_{ij}^{obj}\left[ (x_i-\hat{x}_i)^2 + (y_i-\hat{y}_i)^2 \right]$$ We have that $$(x_i,y_i)$$ are the coordinates of the center of the ground-truth bounding box, and $$(\hat{x}_i,\hat{y}_i)$$ are the predicted ones.

Given this, particularizing to a certain grid cell, $$i$$, and to our predicted bounding box of interest, $$j$$, we have that its contribution to this part of the loss function is: $$\lambda_{coord}\left[ (x_i^{\{0\}}-\hat{x}_i^{\{0\}})^2 + (y_i^{\{0\}}-\hat{y}_i^{\{0\}})^2 + (x_i^{\{1\}}-\hat{x}_i^{\{1\}})^2 + (y_i^{\{1\}}-\hat{y}_i^{\{1\}})^2\right]$$ Where the superscripts $$\{0\}$$ and $$\{1\}$$ represent each ob the ground-truth objects being detected.

As we see, we will be computing this part of the loss using both ground-truth bounding boxes. This makes sense if the objects that are being detected have similar sizes/ aspect ratios and are close to each other (they fall on the same grid cell).

If we focus on the confidence score part of the loss function, where we evaluate the predicted $$\text{IOU}$$ ($$\hat{C}_i$$), with the ground truth $$\text{IOU}$$ ($$C_i$$): $$\sum_{i=0}^{S^2}\sum_{j=0}^B 1_{ij}^{obj}(C_i -\hat{C}_i)^2$$ Then, particularizing to a certain grid cell, $$i$$, and to a certain bounding box, its double contribution will be given by: $$(C_i^{\{0\}} -\hat{C}_i^{\{0\}})^2 + (C_i^{\{1\}} -\hat{C}_i^{\{1\}})^2$$ Where the superscripts $${0}$$ and $${1}$$ have been used again to address this specific situation.

Note: Delving into this question I realized that the previous arguments are just a interpretation of what the algorithm would do in this specific situation. I think that another plausible methodology to follow on this situation would be to just use the ground-truth object with the highest $$\text{IOU}$$.

• Thank you for your answer. If I may get more clarification on this I would be grateful. The problem that arises if one predictor can be related to multiple ground-truth objects is which ground truth object to select when calculating the loss function. – monolith Jan 28 at 11:23
• @monolith I edited the answer addressing this. – Javier TG Feb 3 at 10:44
• Thank you kind Sir for your detailed and quick answer. Only one question more: When calculating the confidence loss in your example, would we add the IoU contribution between this predicted box and both objects, or only the maximum IoU between this one predicted and two ground truth objects? – monolith Feb 3 at 12:55
• @monolith I just updated it. Please read first the final note in the answer. – Javier TG Feb 3 at 14:16