How to make a region of interest proposal from convolutional feature maps?

Question

Problem

Keras does not have any direct implementation of region of interest pooling. I am aware of how to perform maxpooling, but I don't know how to get bounding boxes from feature maps passed from convolutional layer.

Is there any way to directly implement a region proposal algorithm?

---

Example

Let's say there is an architecture like this:

So we have a multi-input neural network architecture that eventually leads to the ROI MaxPool layer. We have three inputs, screenshot, textmaps and candidates, let's take candidates out. Then we would have such code in Keras:

from keras.models import Model
from keras.layers import Input, Dense, Conv2D, ZeroPadding2D, MaxPooling2D, BatchNormalization, concatenate
from keras.activations import relu
from keras.initializers import RandomUniform, Constant, TruncatedNormal

#  Network 1, Layer 1
screenshot = Input(shape=(1280, 1280, 0),
                   dtype='float32',
                   name='screenshot')
# padded1 = ZeroPadding2D(padding=5, data_format=None)(screenshot)
conv1 = Conv2D(filters=96,
               kernel_size=11,
               strides=(4, 4),
               activation=relu,
               padding='same')(screenshot)
# conv1 = Conv2D(filters=96, kernel_size=11, strides=(4, 4), activation=relu, padding='same')(padded1)
pooling1 = MaxPooling2D(pool_size=(3, 3),
                        strides=(2, 2),
                        padding='same')(conv1)
normalized1 = BatchNormalization()(pooling1)  # https://stats.stackexchange.com/questions/145768/importance-of-local-response-normalization-in-cnn

# Network 1, Layer 2

# padded2 = ZeroPadding2D(padding=2, data_format=None)(normalized1)
conv2 = Conv2D(filters=256,
               kernel_size=5,
               activation=relu,
               padding='same')(normalized1)
# conv2 = Conv2D(filters=256, kernel_size=5, activation=relu, padding='same')(padded2)
normalized2 = BatchNormalization()(conv2)
# padded3 = ZeroPadding2D(padding=1, data_format=None)(normalized2)
conv3 = Conv2D(filters=384,
               kernel_size=3,
               activation=relu,
               padding='same',
               kernel_initializer=TruncatedNormal(stddev=0.01),
               bias_initializer=Constant(value=0.1))(normalized2)
# conv3 = Conv2D(filters=384, kernel_size=3, activation=relu, padding='same',
#               kernel_initializer=RandomUniform(stddev=0.1),
#               bias_initializer=Constant(value=0.1))(padded3)

# Network 2, Layer 1

textmaps = Input(shape=(160, 160, 128),
                 dtype='float32',
                 name='textmaps')
txt_conv1 = Conv2D(filters=48,
                   kernel_size=1,
                   activation=relu,
                   padding='same',
                   kernel_initializer=TruncatedNormal(stddev=0.01),
                   bias_initializer=Constant(value=0.1))(textmaps)

# (Network 1 + Network 2), Layer 1

merged = concatenate([conv3, txt_conv1], axis=-1)
merged_padding = ZeroPadding2D(padding=2, data_format=None)(merged)
merged_conv = Conv2D(filters=96,
                     kernel_size=5,
                     activation=relu, padding='same',
                     kernel_initializer=TruncatedNormal(stddev=0.01),
                     bias_initializer=Constant(value=0.1))(merged_padding)

If you look at the end of the code (and architecture itself), we pass concatenated activations from two different Conv+ReLu layers and then pass it to ROI MaxPool layer.

---

Thank you!

Answer 1

To implement region proposal you need two major parts:

• The region proposal network that generates a set of candidate bounding boxes. It can be implemented simply as two convolutional layers to 1) predict the object presense and 2) predict offsets for the default (anchor bounding boxes)

• The ROI pooling layer that provides a fixed-size feature vector for an arbitrary sized proposal.

Here is an implementation of Faster R-CNN in Keras, and here is a detailed explanation of the model and the code.

Here is implementations of the RPN, and here is implementation of the ROI pooling.