# Image Captioning in Keras

I am trying to implement demo of Image Captioning system from Keras documentation. From the documentation I could understand training part.

max_caption_len = 16
vocab_size = 10000

# first, let's define an image model that
# will encode pictures into 128-dimensional vectors.
# it should be initialized with pre-trained weights.
image_model = VGG-16 CNN definition

# next, let's define a RNN model that encodes sequences of words
# into sequences of 128-dimensional word vectors.
language_model = Sequential()

# let's repeat the image vector to turn it into a sequence.

# the output of both models will be tensors of shape (samples, max_caption_len, 128).
# let's concatenate these 2 vector sequences.
model = Merge([image_model, language_model], mode='concat', concat_axis=-1)
# let's encode this vector sequence into a single vector
# which will be used to compute a probability
# distribution over what the next word in the caption should be!

model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)


But now I am confused in how to generate caption for test image. Input here is [image, partial_caption] pair, now for test image how to input partial caption?

• Pl include a link to the documentation in the qn, please :) Would be helpful for the users! – Dawny33 Feb 24 '16 at 4:50
• @Dawny33 Link added. – SHASHANK GUPTA Feb 24 '16 at 4:57
• do u finish your work? sorry for asking, could u send to me your full source code. I just started to learn about keras, so face many problems – сергей михайлов Oct 23 '17 at 0:09

This example trains an image and a partial caption to predict the next word in the caption.

Input: [🐱, "<BEGIN> The cat sat on the"]
Output: "mat"


Notice the model doesn't predict the entire output of the caption only the next word. To construct a new caption, you would have to predict multiple times for each word.

Input: [🐱, "<BEGIN>"] # predict "The"
Input: [🐱, "<BEGIN> The"] # predict "cat"
Input: [🐱, "<BEGIN> The cat"] # predict "sat"
...


To predict the entire sequence, I believe you need to use TimeDistributedDense for the output layer.

Input: [🐱, "<BEGIN> The cat sat on the mat"]
Output: "The cat sat on the mat <END>"


See this issue: https://github.com/fchollet/keras/issues/1029