2
$\begingroup$

Problem

Keras does not have any direct implementation of region of interest pooling. I am aware of how to perform maxpooling, but I don't know how to get bounding boxes from feature maps passed from convolutional layer.

Is there any way to directly implement a region proposal algorithm?


Example

Let's say there is an architecture like this:

enter image description here

So we have a multi-input neural network architecture that eventually leads to the ROI MaxPool layer. We have three inputs, screenshot, textmaps and candidates, let's take candidates out. Then we would have such code in Keras:

from keras.models import Model
from keras.layers import Input, Dense, Conv2D, ZeroPadding2D, MaxPooling2D, BatchNormalization, concatenate
from keras.activations import relu
from keras.initializers import RandomUniform, Constant, TruncatedNormal

#  Network 1, Layer 1
screenshot = Input(shape=(1280, 1280, 0),
                   dtype='float32',
                   name='screenshot')
# padded1 = ZeroPadding2D(padding=5, data_format=None)(screenshot)
conv1 = Conv2D(filters=96,
               kernel_size=11,
               strides=(4, 4),
               activation=relu,
               padding='same')(screenshot)
# conv1 = Conv2D(filters=96, kernel_size=11, strides=(4, 4), activation=relu, padding='same')(padded1)
pooling1 = MaxPooling2D(pool_size=(3, 3),
                        strides=(2, 2),
                        padding='same')(conv1)
normalized1 = BatchNormalization()(pooling1)  # https://stats.stackexchange.com/questions/145768/importance-of-local-response-normalization-in-cnn

# Network 1, Layer 2

# padded2 = ZeroPadding2D(padding=2, data_format=None)(normalized1)
conv2 = Conv2D(filters=256,
               kernel_size=5,
               activation=relu,
               padding='same')(normalized1)
# conv2 = Conv2D(filters=256, kernel_size=5, activation=relu, padding='same')(padded2)
normalized2 = BatchNormalization()(conv2)
# padded3 = ZeroPadding2D(padding=1, data_format=None)(normalized2)
conv3 = Conv2D(filters=384,
               kernel_size=3,
               activation=relu,
               padding='same',
               kernel_initializer=TruncatedNormal(stddev=0.01),
               bias_initializer=Constant(value=0.1))(normalized2)
# conv3 = Conv2D(filters=384, kernel_size=3, activation=relu, padding='same',
#               kernel_initializer=RandomUniform(stddev=0.1),
#               bias_initializer=Constant(value=0.1))(padded3)

# Network 2, Layer 1

textmaps = Input(shape=(160, 160, 128),
                 dtype='float32',
                 name='textmaps')
txt_conv1 = Conv2D(filters=48,
                   kernel_size=1,
                   activation=relu,
                   padding='same',
                   kernel_initializer=TruncatedNormal(stddev=0.01),
                   bias_initializer=Constant(value=0.1))(textmaps)

# (Network 1 + Network 2), Layer 1

merged = concatenate([conv3, txt_conv1], axis=-1)
merged_padding = ZeroPadding2D(padding=2, data_format=None)(merged)
merged_conv = Conv2D(filters=96,
                     kernel_size=5,
                     activation=relu, padding='same',
                     kernel_initializer=TruncatedNormal(stddev=0.01),
                     bias_initializer=Constant(value=0.1))(merged_padding)

If you look at the end of the code (and architecture itself), we pass concatenated activations from two different Conv+ReLu layers and then pass it to ROI MaxPool layer.


Thank you!

$\endgroup$
  • 1
    $\begingroup$ RPN, RoI Pooling/Align, some loss functions are not implemented out-of-the-box. You need to install the Faster/Mask R-CNN library to do this $\endgroup$ – Alex Jan 22 at 23:59
  • $\begingroup$ @Alex Yes I see that they utilize their special loss function (for my architecture I use cross entropy, but here it is different). I guess I will just use architecture in the image above as a base network and on the top of it perform region of interest pooling. I guess I'll just follow Faster R-CNN library code. $\endgroup$ – ShellRox Jan 23 at 9:18
2
$\begingroup$

To implement region proposal you need two major parts:

  • The region proposal network that generates a set of candidate bounding boxes. It can be implemented simply as two convolutional layers to 1) predict the object presense and 2) predict offsets for the default (anchor bounding boxes)

  • The ROI pooling layer that provides a fixed-size feature vector for an arbitrary sized proposal.

Here is an implementation of Faster R-CNN in Keras, and here is a detailed explanation of the model and the code.

Here is implementations of the RPN, and here is implementation of the ROI pooling.

$\endgroup$
  • $\begingroup$ Hello. Thank you for the answer, I've seen the ROI Pooling reference before but I haven't seen the RPN one. Do you know how could it be used for this purpose? Do I pass features of my convolutional network to RPN? (so that it creates region proposals that are later maxpooled). $\endgroup$ – ShellRox Jan 22 at 20:13
  • 1
    $\begingroup$ Right, RPN is usually implemented as a couple of additional layers on the top of your backbone (e.g. VGG) network. From the example: x = Conv2D(512, (3, 3), padding='same', activation='relu', name='rpn_conv1')(base_layers) $\endgroup$ – Dmytro Prylipko Jan 22 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.