I am practicing one of the popular image captioning keras model (LINK IS HERE). Basically this model takes Flickr8k dataset where each image has 5 captions.

1000268201_693b08cb0e.jpg,A child in a pink dress is climbing up a set of stairs in an entry 
way .
1000268201_693b08cb0e.jpg,A girl going into a wooden building .
1000268201_693b08cb0e.jpg,A little girl climbing into a wooden playhouse .
1000268201_693b08cb0e.jpg,A little girl climbing the stairs to her playhouse .
1000268201_693b08cb0e.jpg,A little girl in a pink dress going into a wooden cabin .

During training they feed all five captions with each image as below

for i in range(self.num_captions_per_image):
    loss, acc = self._compute_caption_loss_and_acc(
        img_embed, batch_seq[:, i, :], training=False

when I train this model and try to predict some images from the validation data it generates only one caption with each image sample. While I need to generate the same number of captions per image as we trained the model (i.e 5 captions per image).

I am confused which part of the below code should I change to generate the same number of images as my validation data have.

vocab = vectorization.get_vocabulary()
index_lookup = dict(zip(range(len(vocab)), vocab))
max_decoded_sentence_length = SEQ_LENGTH - 1
valid_images = list(valid_data.keys())

def generate_caption():
  # Select a random image from the validation dataset
  sample_img = np.random.choice(valid_images)

  # Read the image from the disk
  sample_img = decode_and_resize(sample_img)
  img = sample_img.numpy().clip(0, 255).astype(np.uint8)

  # Pass the image to the CNN
  img = tf.expand_dims(sample_img, 0)
  img = caption_model.cnn_model(img)

  # Pass the image features to the Transformer encoder
  encoded_img = caption_model.encoder(img, training=False)

  # Generate the caption using the Transformer decoder
  decoded_caption = "<start> "
  for i in range(max_decoded_sentence_length):
     tokenized_caption = vectorization([decoded_caption])[:, :-1]
     mask = tf.math.not_equal(tokenized_caption, 0)
     predictions = caption_model.decoder(
     tokenized_caption, encoded_img, training=False, mask=mask
     sampled_token_index = np.argmax(predictions[0, i, :])
     sampled_token = index_lookup[sampled_token_index]
     if sampled_token == " <end>":
     decoded_caption += " " + sampled_token

  decoded_caption = decoded_caption.replace("<start> ", "")
  decoded_caption = decoded_caption.replace(" <end>", "").strip()
  print("Predicted Caption: ", decoded_caption)

# Check predictions for a few samples 

enter image description here

Predicted Caption:  a group of dogs race in the snow

The above code generates just one caption from the above image while in the actual dataset this image has 5 captions. I have made a rough image in paint to show what I really want.

enter image description here

Thank You



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.