Faster-RCNN how anchor work with slider in RPN layer?

Question

I am trying to understand the whole Faster-RCNN,

From https://www.quora.com/How-does-the-region-proposal-network-RPN-in-Faster-R-CNN-work

Then a sliding window is run spatially on these feature maps. The size of sliding window is n×n (here 3×3). For each sliding window, a set of 9 anchors are generated which all have the same center (xa,ya)(xa,ya) but with 3 different aspect ratios and 3 different scales as shown below. Note that all these coordinates are computed with respect to the original image.

It is much more clear than other articles for my opion, but still hard to understand how the feature map generate.

I saw another flow pics :

The problem, I write below steps for example:

1. If input is 600x1000x3 pic
2. Through VGG16 convnet , layer 13 output feature map is 40x60x512
3. Use a 3x3 sliding window, generate 1x1x512 feature map ???

Here, how 3x3 sliding window use a set of 9 anchors ???

Sorry, I am really new to object detection and image proccessing.

I only have a little understand about the steps, I known 9 anchor shapes(not the real anchor) are used to generate a lot of anchors(2400*9 in this case).

I can only imagine that use 9 anchor shape to slide the original image to get the all IoU . I don't understand how to use 3x3 sliding window in conv feature map here.

I know how anchors be selected, 2400*9 -> ignore cross-boundary -> 6000 -> apply NMS -> 2000 , in each minibatch, it randomly choose 512 anchors from 2000.

What I can't understand is 3x3 slide with 9 anchor shape . Because from original paper, anchors with is 16, height from 11 to 273 . I don't think it use the 13 layer conv output feature map to calculate IoU . Anchor must be apply in original image, so what is the 3x3 sliding window doing ??

Answer 1

For each anchor you find an IoU with the object in the picture and set 1 if IoUexceeds the threshold and 0 if it is below a lower tjreshold(e.g. 0.3). If it's a hit a bbox offset is calculated, distance between prediction and true bbox. Hence there are two loss fumctions: object/bg and bbox regression

Answer 2

At the core of it, if you look at the source code, RPN is just a convolution layer with the number of maps=number of anchors per location, in you case it's 9. As any convolution layer, it's connected to the previous layer (also convolution, with 256 maps) using a kernel, in your case 3x3. This is exactly what 'Generate 9 anchors for each sliding window on conv. feature map) says. All 9 RPN maps are the same size, so each value $(i,j)$ in each feature map is the score of the corresponding anchor for that location $(i,j)$. Another convlayer with $9x4$ feature maps is also created for every anchor to predict bounding box offsets. Values in these two convlayers are obtained at feedforward stage.

Since sizes and aspect ratios of all anchors are different, AnchorGenerator needs to compute their actual parameters $(x,y,h,w)$. To do this, it takes the image size, output feature layer and anchor hyperparameters. It maps the feature map size to the image size to get a cell grid, and derives anchor parameters for each cell (location). So now you have all you need to compute loss: ground truth (class + box coordinates), anchor coordinates and score+box offset prediction. For anchors that overlap with the object>threshold (anchor coords vs gt box coords), RPN takes from the corresponding feature map in the first conv layer the score prediction and 4 predicted box offsets from the second conv layer. The score is used in a binary cross-entropy loss, the box prediction in MSE-type loss (e.g. Smooth1Loss). Anchor outputs offset prediction (box-anchor), the gt label is the offset between gt box and anchor.

Done!

Now, box predictions are inputs in the next layer (RoIAlign).

Answer 3

A pixels does map to 9 anchors. But the 3 by 3 convolutional layer before 1 by 1 maybe aim to extra non-linear(the extra 3 by 3 con layer produce same width and height as input).

So, not 9 pixels map to 9 anchors.

Here is a video from CS231n, shows the above idea at 51min.

Answer 4

The 3*3 sliding window is used to calculate the regression and classification loss of the RPN network which determines the bounding box and also whether it contains object or not!