I am practicing one of the popular image captioning keras model (LINK IS HERE). Basically this model takes Flickr8k dataset where each image has 5 captions.
1000268201_693b08cb0e.jpg,A child in a pink dress is climbing up a set of stairs in an entry way . 1000268201_693b08cb0e.jpg,A girl going into a wooden building . 1000268201_693b08cb0e.jpg,A little girl climbing into a wooden playhouse . 1000268201_693b08cb0e.jpg,A little girl climbing the stairs to her playhouse . 1000268201_693b08cb0e.jpg,A little girl in a pink dress going into a wooden cabin .
During training they feed all five captions with each image as below
for i in range(self.num_captions_per_image): loss, acc = self._compute_caption_loss_and_acc( img_embed, batch_seq[:, i, :], training=False )
when I train this model and try to predict some images from the validation data it generates only one caption with each image sample. While I need to generate the same number of captions per image as we trained the model (i.e 5 captions per image).
I am confused which part of the below code should I change to generate the same number of images as my validation data have.
vocab = vectorization.get_vocabulary() index_lookup = dict(zip(range(len(vocab)), vocab)) max_decoded_sentence_length = SEQ_LENGTH - 1 valid_images = list(valid_data.keys()) def generate_caption(): # Select a random image from the validation dataset sample_img = np.random.choice(valid_images) # Read the image from the disk sample_img = decode_and_resize(sample_img) img = sample_img.numpy().clip(0, 255).astype(np.uint8) plt.imshow(img) plt.show() # Pass the image to the CNN img = tf.expand_dims(sample_img, 0) img = caption_model.cnn_model(img) # Pass the image features to the Transformer encoder encoded_img = caption_model.encoder(img, training=False) # Generate the caption using the Transformer decoder decoded_caption = "<start> " for i in range(max_decoded_sentence_length): tokenized_caption = vectorization([decoded_caption])[:, :-1] mask = tf.math.not_equal(tokenized_caption, 0) predictions = caption_model.decoder( tokenized_caption, encoded_img, training=False, mask=mask ) sampled_token_index = np.argmax(predictions[0, i, :]) sampled_token = index_lookup[sampled_token_index] if sampled_token == " <end>": break decoded_caption += " " + sampled_token decoded_caption = decoded_caption.replace("<start> ", "") decoded_caption = decoded_caption.replace(" <end>", "").strip() print("Predicted Caption: ", decoded_caption) # Check predictions for a few samples generate_caption()
Predicted Caption: a group of dogs race in the snow
The above code generates just one caption from the above image while in the actual dataset this image has 5 captions. I have made a rough image in paint to show what I really want.