I have a dataset of memes, and I'm trying to predict if a certain meme is sexist or not, using image and text together.

Right now I have two models, a VGG16 fine tuned CNN for images and a LSTM model for text, each of them with Keras.

Both models perform well alone (~0.8-0.9 accuracy), and I'm trying to merge them to see if I can get a better result.

I'm concatenating the output of each model like this:


input_tensor = layers.Input(shape=(image_size,image_size,3))

vgg_model = VGG16(input_tensor = input_tensor, weights = 'imagenet', include_top=False)

for layer in vgg_model.layers:
    layer.trainable = False

l2_strength = 1e-5
dropout_prob = 0.5

x = vgg_model.output
x = layers.Flatten(input_shape=vgg_model.output_shape[1:])(x)
x = layers.Dense(256, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x)
x = layers.Dropout(dropout_prob)(x)
x = layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x)

vgg_model = models.Model(vgg_model.input, x)


input_text = layers.Input(shape=(1, 512))
x = layers.LSTM(32, dropout=0.5, name="LTSM")(input_text)
x = layers.Dense(10, kernel_initializer='normal', activation='relu')(x)

text_model = models.Model(input_text, x)

x = layers.concatenate([vgg_model.output, text_model.output])
out = layers.Dense(1, activation='softmax', name='output_layer')(x)

merged_model = models.Model([vgg_model.input, text_model.input], out)


and I train the model like this:

    [train_image, train_text], 
    y = train_y, 

where train_images are the images and train_text are the associated text, and the label are a 1 or 0 if a meme is sexist or not.

The problem is that both train and validation accuracy are always 0.5, with a loss of ~8.0-9.0, and never improve during training, so it's not able to learn, but it's weird seen that the model alone give good performance.

I was wondering if someone encountered the same problem and/or do you have any suggestion on why this happen.


I had the same problem and I solved it adding a BatchNormalization layer. For example:

# create model ConvLSTM
input_convlstm = Input(name='convlstm', shape=(n_steps, 1, n_length, n_metadata))
branch_convlstm = BatchNormalization()(input_convlstm)
branch_convlstm = ConvLSTM2D(filters=64, kernel_size=(1,3), activation='tanh', input_shape=(n_steps, 1, n_length, n_metadata))(branch_convlstm)
branch_convlstm = Flatten()(branch_convlstm)
branch_convlstm = Dense(128, activation='tanh')(branch_convlstm)

# create model metadata
input_metadata = Input(name='metadata', shape=(21,))
branch_metadata = Dense(neurons[0], activation='tanh')(input_metadata)
branch_metadata = BatchNormalization()(branch_metadata)
branch_metadata = Dropout(dropout_rate)(branch_metadata)

# create final model
concat = Concatenate()([branch_convlstm, branch_metadata])
out = Dense(n_outputs, activation='softmax')(concat)
model = Model(inputs=[input_convlstm, input_metadata], outputs=out)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.