I have a dataset of memes, and I'm trying to predict if a certain meme is sexist or not, using image and text together.
Right now I have two models, a
VGG16 fine tuned CNN for images and a
LSTM model for text, each of them with
Both models perform well alone (~0.8-0.9 accuracy), and I'm trying to merge them to see if I can get a better result.
I'm concatenating the output of each model like this:
CNN input_tensor = layers.Input(shape=(image_size,image_size,3)) vgg_model = VGG16(input_tensor = input_tensor, weights = 'imagenet', include_top=False) for layer in vgg_model.layers: layer.trainable = False l2_strength = 1e-5 dropout_prob = 0.5 x = vgg_model.output x = layers.Flatten(input_shape=vgg_model.output_shape[1:])(x) x = layers.Dense(256, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x) x = layers.Dropout(dropout_prob)(x) x = layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x) vgg_model = models.Model(vgg_model.input, x)
LSTM input_text = layers.Input(shape=(1, 512)) x = layers.LSTM(32, dropout=0.5, name="LTSM")(input_text) x = layers.Dense(10, kernel_initializer='normal', activation='relu')(x) text_model = models.Model(input_text, x)
MERGED x = layers.concatenate([vgg_model.output, text_model.output]) out = layers.Dense(1, activation='softmax', name='output_layer')(x) merged_model = models.Model([vgg_model.input, text_model.input], out) merged_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
and I train the model like this:
merged_model.fit( [train_image, train_text], y = train_y, epochs=50, batch_size=64, validation_split=0.2)
train_images are the images and
train_text are the associated text, and the label are a 1 or 0 if a meme is sexist or not.
The problem is that both
validation accuracy are always 0.5, with a loss of ~8.0-9.0, and never improve during training, so it's not able to learn, but it's weird seen that the model alone give good performance.
I was wondering if someone encountered the same problem and/or do you have any suggestion on why this happen.