Merge two vgg16 models in Keras

Question

I am working in deep learning project using vgg16. I got the following error and I could not solve it.

RuntimeError: ('The name "block1_conv1" is used 2 times in the model. All layer names should be unique.

The code:

vgg16_model = keras.applications.vgg16.VGG16()

input_layer1 = vgg16_model.input
vgg16_model.get_layer(index = 0).name = 'input1'
last_layer1 = vgg16_model.get_layer('flatten').output

# Adding layers
x = Dense(2048, activation='relu', name='fc1')(last_layer1)
x = BatchNormalization(name='bn1')(x)
fc12 = Dense(2048, activation='relu', name='fc2')(x)
bn12 = BatchNormalization(name='bn2')(fc12)
out1 = Dropout(0.4, name='dropout1')(bn12)

#Classification layer
output_layer1 = Dense(no_classes, activation='softmax', name='prediction1')(out1) #106 classes
model1 = Model(input_layer1, output_layer1)

#Train the last block
model1.trainable = True
set_trainable = False
for layer in model1.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

vgg16_model2 = keras.applications.vgg16.VGG16()

input_layer2 = vgg16_model2.input
vgg16_model2.get_layer(index = 0).name = 'input2'

# Remove layers from block4_pool layer
last_layer2 = vgg16_model2.get_layer('block4_pool').output

# Adding layers
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv1', padding='same')(last_layer2)
x = BatchNormalization(name='bn1')(x)
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv2', padding='same')(x)
x = BatchNormalization(name='bn2')(x)
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv3', padding='same')(x)
x = BatchNormalization(name='bn3')(x)
x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), name = 'block5_pool')(x)
x = Flatten(name = 'flatten')(x)
x = Dense(2048, activation='relu', name='fc1')(x)
x = BatchNormalization(name='bn4')(x)
fc22 = Dense(2048, activation='relu', name='fc2' )(x)
bn25 = BatchNormalization(name='bn5')(fc22)
out2 = Dropout(0.3, name='dropout1')(bn25)

#Classification layer
output_layer2 = Dense(106, activation='softmax', name='prediction2')(out2) 
model2 = Model(input_layer2, output_layer2)

#Train the last block
model2.trainable = True
set_trainable = False
for layer in model2.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

con = concatenate([out1, out2])  # merge the outputs of the two models
output_layer3 = Dense(no_classes, activation='softmax', name='prediction3')(con)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer3])

I tested model1 and model2, and I got good results. But when I tried to merge the two models the error appear. The problem initiated here:

multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer3])

Please help to solve this problem.

Answer 1

The problem is, you are trying to instantiate two VGG16 models at the same time and its confusing for the kernel to figure out which graph it needs to use. Atleast that is the speculation I have, because of the problems faced when trying to load multiple models in keras with tensorflow backend, its not straight forward.

Try commenting the code from when you are instantiating the second model and this error should go away.