# Keras: Merged/Concatenated model perform worst than separate models memes recognition

I have a dataset of memes, and I'm trying to predict if a certain meme is sexist or not, using image and text together.

Right now I have two models, a VGG16 fine tuned CNN for images and a LSTM model for text, each of them with Keras.

Both models perform well alone (~0.8-0.9 accuracy), and I'm trying to merge them to see if I can get a better result.

I'm concatenating the output of each model like this:

CNN

input_tensor = layers.Input(shape=(image_size,image_size,3))

vgg_model = VGG16(input_tensor = input_tensor, weights = 'imagenet', include_top=False)

for layer in vgg_model.layers:
layer.trainable = False

l2_strength = 1e-5
dropout_prob = 0.5

x = vgg_model.output
x = layers.Flatten(input_shape=vgg_model.output_shape[1:])(x)
x = layers.Dense(256, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x)
x = layers.Dropout(dropout_prob)(x)
x = layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x)

vgg_model = models.Model(vgg_model.input, x)


LSTM

input_text = layers.Input(shape=(1, 512))
x = layers.LSTM(32, dropout=0.5, name="LTSM")(input_text)
x = layers.Dense(10, kernel_initializer='normal', activation='relu')(x)

text_model = models.Model(input_text, x)

MERGED

x = layers.concatenate([vgg_model.output, text_model.output])
out = layers.Dense(1, activation='softmax', name='output_layer')(x)

merged_model = models.Model([vgg_model.input, text_model.input], out)

merged_model.compile(loss='binary_crossentropy',
metrics=['accuracy'])


and I train the model like this:

merged_model.fit(
[train_image, train_text],
y = train_y,
epochs=50,
batch_size=64,
validation_split=0.2)


where train_images are the images and train_text are the associated text, and the label are a 1 or 0 if a meme is sexist or not.

The problem is that both train and validation accuracy are always 0.5, with a loss of ~8.0-9.0, and never improve during training, so it's not able to learn, but it's weird seen that the model alone give good performance.

I was wondering if someone encountered the same problem and/or do you have any suggestion on why this happen.

I had the same problem and I solved it adding a BatchNormalization layer. For example:

# create model ConvLSTM
input_convlstm = Input(name='convlstm', shape=(n_steps, 1, n_length, n_metadata))
branch_convlstm = BatchNormalization()(input_convlstm)
branch_convlstm = ConvLSTM2D(filters=64, kernel_size=(1,3), activation='tanh', input_shape=(n_steps, 1, n_length, n_metadata))(branch_convlstm)
branch_convlstm = Flatten()(branch_convlstm)
branch_convlstm = Dense(128, activation='tanh')(branch_convlstm)