Faster RCNN-RPN NETWORK

Question

I already asked this question in stack overflow, but got response from experts, to post this question here, please help me to understand this concept...

I am trying to understand RPN network in Faster RCNN.

I understand the concept of RPN network,

1. Pass the input images to the pre trained CNN, and get the output as feature maps
2. Make fixed size of the feature maps
3. Extract anchors (3 different scales and ratio for every sliding window) from the fixed size feature maps.
4. Use two 1×1 Fully connected NN to find the background or object and the bounding box coordinates (4 values)
5. Calculate IOU for Anchors bounding box with Ground Truth bounding box, if IOU>0.7, then the anchor has object, otherwise, the anchor has background.

The theme for RPN is to give the region proposals which have objects.

But, I do not understand the input and the output structure.

For example, I have 50 images, each images having 5 to 6 objects, and labeling informations(coordinates of each objects).

How do I generate target values, to train PRN Network...

In all the blogs, they shows the architecture as feed the entire image to the pre trained CNN.

And, the output of RPN, the model has to tell whether the anchor has object or not, and also predict the bounding box for the object in the anchor.

For this, how to prepare the input and target/output values like we do in dog/cat or dog/cat/car classification problem.

Let me correct if I am not correct,

Is that, we have to crop all the objects in every image and do binary classification as object vs background for classifying the anchor has object or not

And, Is that, we have to give the ground truth value as target for every cropped objects from all images in the dataset, so that the RPN network trained well to predict the bounding box for the object in every anchor.

Hope, I clearly explained my doubts.

Help me to learn this concept, Thank you

Answer 1

Is that, we have to crop all the objects in every image and do binary classification as object vs background for classifying the anchor has object or not

The RPN gets the input from backbone network(VGG, Resnet etc.) as feature maps. Here the RPN itself a CNN layer so it will handle different shaped anchors to FC layer.

For the loss calculation, each feature map points will mapped to original image point (reverse scaling) and calculate the IOU score for loss function. Possible the scaling will be straight forward based on the stripes and padding.

From these the RPN will learn to choose the bboxes.

In Faster RCNN there are different implementation methods between RPN and object classification layers.

If you are familiar with TF, you can refer this implementation. https://github.com/tensorflow/models/blob/master/research/object_detection/meta_architectures/faster_rcnn_meta_arch.py