# Problem

Keras does not have any direct implementation of region of interest pooling. I am aware of how to perform maxpooling, but I don't know how to get bounding boxes from feature maps passed from convolutional layer.

Is there any way to directly implement a region proposal algorithm?

# Example

Let's say there is an architecture like this:

So we have a multi-input neural network architecture that eventually leads to the ROI MaxPool layer. We have three inputs, screenshot, textmaps and candidates, let's take candidates out. Then we would have such code in Keras:

from keras.models import Model
from keras.layers import Input, Dense, Conv2D, ZeroPadding2D, MaxPooling2D, BatchNormalization, concatenate
from keras.activations import relu
from keras.initializers import RandomUniform, Constant, TruncatedNormal

#  Network 1, Layer 1
screenshot = Input(shape=(1280, 1280, 0),
dtype='float32',
name='screenshot')
conv1 = Conv2D(filters=96,
kernel_size=11,
strides=(4, 4),
activation=relu,
pooling1 = MaxPooling2D(pool_size=(3, 3),
strides=(2, 2),
normalized1 = BatchNormalization()(pooling1)  # https://stats.stackexchange.com/questions/145768/importance-of-local-response-normalization-in-cnn

# Network 1, Layer 2

conv2 = Conv2D(filters=256,
kernel_size=5,
activation=relu,
normalized2 = BatchNormalization()(conv2)
conv3 = Conv2D(filters=384,
kernel_size=3,
activation=relu,
kernel_initializer=TruncatedNormal(stddev=0.01),
bias_initializer=Constant(value=0.1))(normalized2)
# conv3 = Conv2D(filters=384, kernel_size=3, activation=relu, padding='same',
#               kernel_initializer=RandomUniform(stddev=0.1),

# Network 2, Layer 1

textmaps = Input(shape=(160, 160, 128),
dtype='float32',
name='textmaps')
txt_conv1 = Conv2D(filters=48,
kernel_size=1,
activation=relu,
kernel_initializer=TruncatedNormal(stddev=0.01),
bias_initializer=Constant(value=0.1))(textmaps)

# (Network 1 + Network 2), Layer 1

merged = concatenate([conv3, txt_conv1], axis=-1)
merged_conv = Conv2D(filters=96,
kernel_size=5,
kernel_initializer=TruncatedNormal(stddev=0.01),


If you look at the end of the code (and architecture itself), we pass concatenated activations from two different Conv+ReLu layers and then pass it to ROI MaxPool layer.

Thank you!

• RPN, RoI Pooling/Align, some loss functions are not implemented out-of-the-box. You need to install the Faster/Mask R-CNN library to do this – Alex Jan 22 at 23:59
• @Alex Yes I see that they utilize their special loss function (for my architecture I use cross entropy, but here it is different). I guess I will just use architecture in the image above as a base network and on the top of it perform region of interest pooling. I guess I'll just follow Faster R-CNN library code. – ShellRox Jan 23 at 9:18

• Right, RPN is usually implemented as a couple of additional layers on the top of your backbone (e.g. VGG) network. From the example: x = Conv2D(512, (3, 3), padding='same', activation='relu', name='rpn_conv1')(base_layers) – Dmytro Prylipko Jan 22 at 20:45