Backpropagation in Faster R-CNN

Question

I understand how the convolution layers are applied after selective search finds the regions of interest in vanilla R-CNN and so the back-propagation or any weight updating is done in the individual convolution networks. But in Faster R-CNN a convNet is applied first and then RPN works out the regions of interest in a separate convNet. Where exactly are the weights updated? Is it between the initial convNet and the RPN. But weights can only be updated after comparison with the ground truth values. Or is it that after a full forward pass of both networks, a backward pass is done. In this case how are both considered simultaneously? If not, Where and how is this comparison done? Any pointers to any resource will be highly appreciated.

Answer 1

Where exactly are the weights updated? Is it between the initial convNet and the RPN. All updateable weights are updated during backpropagation.

I assume you were trying to ask when are the weights updated?, well according to the original paper there are 3 way to train Faster R-CNN, each has it's own when to update weight time:

1. Alternating training aka 4-step alternating training, which is the one used by the author in the original faster r-cnn paper. It's explained quite well in the original paper
2. Approximate joint training, the one you mention in the question, after a full forward pass of both networks, a backward pass is done. I'm not quite sure what you mean by "In this case how are both considered simultaneously?"
3. Non-approximate joint training, I don't really understand about this one

Back to the Where exactly are the weights updated? question, well, some to be mention are the RPN kernerls, RPN Cls and Reg FC Layer, the detection (Fast R-CNN) Cls and Reg FC layer also the CNN kernels (if you didn't freeze them)

I hope this help since I'm also learning about Faster R-CNN too, also sorry for the bad English.

Answer 2

If you use the torchvision implementation, I suggest you have a look at the source code, e.g. on Github. RPN and RoI losses (boxes, classes, masks) are combined after the feedforward step. They are accumulated (summed), then backpropagated starting at the last layer of RoI, then through the RPN, and then through FPN and the backbone.