# How to train two neural networks together

This could be considered as an extension of my previous question "How to make a region of interest proposal from convolutional feature maps?".

# Network 1:

I have a multi-input neural network, it takes three types of inputs:

Screenshot: 1280x800 2D input shape.

TextMaps: 160x160x168 input shape.

Candidates: just bounding boxes of regions (not relevant in this case).

As displayed in the question above, this is the image of architecture:

And here's the code:

from keras.models import Model
from keras.layers import Input, Dense, Conv2D, ZeroPadding2D, MaxPooling2D, BatchNormalization, concatenate
from keras.activations import relu
from keras.initializers import RandomUniform, Constant, TruncatedNormal

#  Network 1, Layer 1
screenshot = Input(shape=(1280, 1280, 0),
dtype='float32',
name='screenshot')
conv1 = Conv2D(filters=96,
kernel_size=11,
strides=(4, 4),
activation=relu,
pooling1 = MaxPooling2D(pool_size=(3, 3),
strides=(2, 2),
normalized1 = BatchNormalization()(pooling1)  # https://stats.stackexchange.com/questions/145768/importance-of-local-response-normalization-in-cnn

# Network 1, Layer 2

conv2 = Conv2D(filters=256,
kernel_size=5,
activation=relu,
normalized2 = BatchNormalization()(conv2)
conv3 = Conv2D(filters=384,
kernel_size=3,
activation=relu,
kernel_initializer=TruncatedNormal(stddev=0.01),
bias_initializer=Constant(value=0.1))(normalized2)
# conv3 = Conv2D(filters=384, kernel_size=3, activation=relu, padding='same',
#               kernel_initializer=RandomUniform(stddev=0.1),

# Network 2, Layer 1

textmaps = Input(shape=(160, 160, 128),
dtype='float32',
name='textmaps')
txt_conv1 = Conv2D(filters=48,
kernel_size=1,
activation=relu,
kernel_initializer=TruncatedNormal(stddev=0.01),
bias_initializer=Constant(value=0.1))(textmaps)

# (Network 1 + Network 2), Layer 1

merged = concatenate([conv3, txt_conv1], axis=-1)
merged_conv = Conv2D(filters=96,
kernel_size=5,
kernel_initializer=TruncatedNormal(stddev=0.01),


# Problem:

Before I go to a Network 2, I'll already present a problem.

As you see in the image above, at the end of the architecture we have a ROI MaxPool layer. I use a method used presented by Faster R-CNN, which is based on region proposal networks that should be trained alone. Let's refer to region proposal network as Network 2.

But here's the problem, in order to train my Network 1, I need to train Network 2, but in order to train Network 2, I need to train a Network 1.

Is there any way to get beyond this problem?

P. S

Network 2 is nothing more than a network that has 2 convolutional and 1 linear layers, and it is based on feature maps given by Network 1. But it must be converted to region proposals by a special function, hence it must be evaluated during the training (but I can't just pass an empty shape information, it requires an actual feature map).

If there's any way to get around this, it would be greatly appreciated.

Thank you!

• Did you check merge functionality in Keras? – Aditya Jan 25 '19 at 6:53
• @Aditya I've used concatenate layer at the end. Did you mean keras.layers.Add? If so how can I use it for this purpose? – ShellRox Jan 25 '19 at 7:05