I am trying to implement demo of Image Captioning system from Keras documentation. From the documentation I could understand training part.

max_caption_len = 16
vocab_size = 10000

# first, let's define an image model that
# will encode pictures into 128-dimensional vectors.
# it should be initialized with pre-trained weights.
image_model = VGG-16 CNN definition

# next, let's define a RNN model that encodes sequences of words
# into sequences of 128-dimensional word vectors.
language_model = Sequential()
language_model.add(Embedding(vocab_size, 256, input_length=max_caption_len))
language_model.add(GRU(output_dim=128, return_sequences=True))

# let's repeat the image vector to turn it into a sequence.

# the output of both models will be tensors of shape (samples, max_caption_len, 128).
# let's concatenate these 2 vector sequences.
model = Merge([image_model, language_model], mode='concat', concat_axis=-1)
# let's encode this vector sequence into a single vector
model.add(GRU(256, 256, return_sequences=False))
# which will be used to compute a probability
# distribution over what the next word in the caption should be!

model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)

But now I am confused in how to generate caption for test image. Input here is [image, partial_caption] pair, now for test image how to input partial caption?

  • $\begingroup$ Pl include a link to the documentation in the qn, please :) Would be helpful for the users! $\endgroup$ – Dawny33 Feb 24 '16 at 4:50
  • 1
    $\begingroup$ @Dawny33 Link added. $\endgroup$ – SHASHANK GUPTA Feb 24 '16 at 4:57
  • $\begingroup$ do u finish your work? sorry for asking, could u send to me your full source code. I just started to learn about keras, so face many problems $\endgroup$ – сергей михайлов Oct 23 '17 at 0:09

This example trains an image and a partial caption to predict the next word in the caption.

Input: [🐱, "<BEGIN> The cat sat on the"]
Output: "mat"

Notice the model doesn't predict the entire output of the caption only the next word. To construct a new caption, you would have to predict multiple times for each word.

Input: [🐱, "<BEGIN>"] # predict "The"
Input: [🐱, "<BEGIN> The"] # predict "cat"
Input: [🐱, "<BEGIN> The cat"] # predict "sat"

To predict the entire sequence, I believe you need to use TimeDistributedDense for the output layer.

Input: [🐱, "<BEGIN> The cat sat on the mat"]
Output: "The cat sat on the mat <END>"

See this issue: https://github.com/fchollet/keras/issues/1029

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.