I studied the research paper on Perceptual Generative Adversarial Nets for small object detection.

There they have detailed the structure of Generator network as given in the picture below: enter image description here

I am new in the field of GAN. I am having problem in designing the generator function. Can anyone help me out?


The generator function is just a CNN that maps an image to a feature map here (well, ROI pooled anyway). There is nothing specific to it being a GAN; what makes something a GAN is its use as a density estimator and how it is trained (i.e., adversarially), not its architecture. So don't worry about the lack of familiarity with GANs.

In any case, the complex part here, in my opinion, is the presence of the object detection framework parts. To better understand it, though you probably have done so already, I suggest reading the "RCNN trilogy papers": RCNN, Fast RCNN, and Faster RCNN, or at least the middle one, which is relevant here. See also this post and this one about ROI pooling, which takes the dis-similarly sized ROIs and maps each of them to a small fixed-size feature map.

The rest of the architecture are standard convolutional layers mostly, with which you are presumably familiar. It might be useful to review ResNet though, since ResBlocks are used here. The core ("backbone") of the pipeline relies on the VGG network; its use is explained in more detail in references 24 and 41 of the paper.

Anyway, here's what's happening. Given an image $I$ with proposals $P$ (where $n=|P|$), we first run conv1 on $I$, giving us the low-level feature map $L$. Two things then happen:

  • On the "top path", $L$ is run through convi for i=2:5, resulting in a new feature map $L_T$, which is then ROI pooled via $P$. This gives us a stack of proposals $S_1,\ldots,S_{n}$, each of which is a featurized image.

  • On the "bottom path", $L$ is run through two small conv layers and then ROI pooled as well, giving a similar proposal stack $f_1,\ldots,f_{n}$. These are then processed by $B$ standard ResBlocks, to give us $n$ residuals $r_1,\ldots,r_n$.

Finally, we get the ultimate output of the generator, which is a stack of super-resolved proposals given by $$ Y = (y_1,\ldots,y_n) = ( S_1 \oplus r_1,\ldots, S_n \oplus r_n ) $$ where $\oplus$ here just means element-wise sum (i.e., S_k + r_k in most array-based languages).

Basically, the problem is that the ROI pooled features will "look nice" for big objects but terrible for small ones, due to the fewer number of pixels in the latter. The fix for this here is to do "standard" object detection in the top path, but learn a "residue" $r_j$ in the bottom path for each ROI, so that when this residue is added to a small proposal $S_j$, it is given the details normally only present in large proposals. In other words, we are basically doing super-resolution on the small ROIs to help the detector on small objects.

As the paper authors write in the caption:

The generator is a deep residual network which takes the features with fine-grained details from lower-level layer as input and passes them to 3 × 3 convolutional filters followed by 1 × 1 convolutional filters to increase the feature dimension to be aligned with that of “Conv5”. Then B residual blocks each of which consists of convolutional layers followed by batch normalization and ReLU activation are employed to learn the residual representation, which is used to enhance the pooled features from “Conv5” for small objects to super-resolved representation through element-wise sum operation.

As well as in the main paper:

As shown in Figure 3, the generator takes the feature from the bottom convolutional layer as the input that preserves many low-level details and is informative for feature super-resolution. The resulting feature is first passed into the 3 × 3 convolution filters followed by the 1 × 1 convolution filters to increase the feature dimension to be the same as that of “Conv5”. Then, B residual blocks with the identical layout consisting of two 3×3 convolutional filters followed by batch-normalization layer and ReLU activation layer are introduced to learn the residual representation between the large and the small objects, as a generative model. The learned residual representation is then used to enhance the feature pooled from “Conv5” for the small object proposal through RoI pooling [11] by element-wise sum operation, producing super-resolved representation.

  • $\begingroup$ Hi @user3658307 your answer really helped me understanding the model. Yet I am facing problems in the ROI pooling layer, which I have generated a question https://datascience.stackexchange.com/questions/54735/issues-related-to-the-code-for-roi-pooling-from-the-feature-map $\endgroup$
    – Excelsior
    Jun 29 '19 at 9:38
  • $\begingroup$ @Excelsior Did you look at the RCNN papers? They explain ROI pooling. I suggest using the publicly available implementations of ROI pooling (e.g., look in the github for Fast-RCNN; it should have an ROI pooling implementation). Also, you could simply use the VGGCNN-M-1024 model directly as the paper does. $\endgroup$ Jun 29 '19 at 10:27
  • $\begingroup$ They have used the pretrained VGGCNN-M-1024 model for initializing the network. What does that actually mean? Is this replacing the RoI Pooling layer of the generator model? Though it seems that the RoI pooling layer does the work of extracting the region proposals from an image, which is clearly the idea behind the Faster RCNN paper. Accordingly, I started implementing the Faster R-CNN paper for extracting region proposals with the help of publicly available repositories. But I ran into several bugs. Do you know/can you provide any such code for doing the same? $\endgroup$
    – Excelsior
    Jun 29 '19 at 11:19
  • $\begingroup$ Hey @user3658307 If it is so, can you tell me where I can get pretrained model of VGGCNN-M-1024 that is compatible with keras $\endgroup$
    – Excelsior
    Jun 29 '19 at 11:28
  • $\begingroup$ @Excelsior It means the backbone of the network (those 5 conv layers) is pretrained (on imagenet usually). Initializing the network means the pretrained model is the starting point for those layers. This is unrelated to ROI pooling. The backbone computes the feature maps over which the ROI pooling is done. Also be careful when you say "extracting the region proposals" since that is usually considered the job of selective search or the Region Proposal Network, not that of ROI pooling. But in some sense, yes, ROI pooling normalizes and extracts proposals based on their outputs. $\endgroup$ Jun 29 '19 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.