0
$\begingroup$

I am practicing one of the popular image captioning keras model (LINK IS HERE). Basically this model takes Flickr8k dataset where each image has 5 captions.

1000268201_693b08cb0e.jpg,A child in a pink dress is climbing up a set of stairs in an entry 
way .
1000268201_693b08cb0e.jpg,A girl going into a wooden building .
1000268201_693b08cb0e.jpg,A little girl climbing into a wooden playhouse .
1000268201_693b08cb0e.jpg,A little girl climbing the stairs to her playhouse .
1000268201_693b08cb0e.jpg,A little girl in a pink dress going into a wooden cabin .

During training they feed all five captions with each image as below

for i in range(self.num_captions_per_image):
    loss, acc = self._compute_caption_loss_and_acc(
        img_embed, batch_seq[:, i, :], training=False
    )

when I train this model and try to predict some images from the validation data it generates only one caption with each image sample. While I need to generate the same number of captions per image as we trained the model (i.e 5 captions per image).

I am confused which part of the below code should I change to generate the same number of images as my validation data have.

vocab = vectorization.get_vocabulary()
index_lookup = dict(zip(range(len(vocab)), vocab))
max_decoded_sentence_length = SEQ_LENGTH - 1
valid_images = list(valid_data.keys())


def generate_caption():
  # Select a random image from the validation dataset
  sample_img = np.random.choice(valid_images)

  # Read the image from the disk
  sample_img = decode_and_resize(sample_img)
  img = sample_img.numpy().clip(0, 255).astype(np.uint8)
  plt.imshow(img)
  plt.show()

  # Pass the image to the CNN
  img = tf.expand_dims(sample_img, 0)
  img = caption_model.cnn_model(img)

  # Pass the image features to the Transformer encoder
  encoded_img = caption_model.encoder(img, training=False)

  # Generate the caption using the Transformer decoder
  decoded_caption = "<start> "
  for i in range(max_decoded_sentence_length):
     tokenized_caption = vectorization([decoded_caption])[:, :-1]
     mask = tf.math.not_equal(tokenized_caption, 0)
     predictions = caption_model.decoder(
     tokenized_caption, encoded_img, training=False, mask=mask
     )
     sampled_token_index = np.argmax(predictions[0, i, :])
     sampled_token = index_lookup[sampled_token_index]
     if sampled_token == " <end>":
      break
     decoded_caption += " " + sampled_token

  decoded_caption = decoded_caption.replace("<start> ", "")
  decoded_caption = decoded_caption.replace(" <end>", "").strip()
  print("Predicted Caption: ", decoded_caption)


# Check predictions for a few samples 
generate_caption()

enter image description here

Predicted Caption:  a group of dogs race in the snow

The above code generates just one caption from the above image while in the actual dataset this image has 5 captions. I have made a rough image in paint to show what I really want.

enter image description here

Thank You

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.