I need to train a system on a large set of images and associated captions to determine which (image, caption) pairs are correct and which are not. I don't have any labeled pairs, but I can assume that more pairs are correct than are not. Furthermore, the images and captions have small subsets that are reasonably similar, but not exactly.

Current idea

My current idea is to build a sort of GAN architecture where I have a generator and a discriminator, except that the generator is simply sampling from the existing corpus of images and captions to generate false pairs and the discriminator is in charge of determining which pairs are correct or not.

images, captions = [], []
bpg = bad_pair_generator(images, captions)

clfr = classifier()

for epoch in range(epochs):
    good_pairs, bad_pairs = zip(images, captions), bpg.generate(N)
    X, y = good_pairs + bad_pairs, [1]*len(good_pairs) + [0]*len(bad_pairs)

    clfr.train(X, y)

    trickiness = [yi == yi_pred for yi, yi_pred in zip(y, clfr.predict(X))]
    bpg.fit(bad_pairs, trickiness[-len(bad_pairs):])

After this, I would hope that the classifier was stressed sufficiently to learn a good model for whether the pairs are good or not.


  1. Are there better researched approaches?
  2. If not, does this approach seem sane and reasonably safe?
  3. What kind of model could I use to compared the image to the text? Clearly a CNN/RNN would be good for the image/text, but how do you combine them?

1 Answer 1


A thought:

Maybe an approach like Show and Tell: A neural image caption generator would be useful to look at. There they use exactly as you say a combination of CNN (for image) features and LSTM (for text) + search to generate text given the features from the image.

It is possible that then a discriminator can be used to learn whether 'generated text' is close enough to 'given label' in order to give the final result.

It is also possible that you do not need the discriminator, and instead of using the output of the show and tell network directly (like removing the last linear layer) and feed one of the earlier steps (which represents a probability distribution of possible texts to generate) into the discriminator so that it has more possible combinations of text outputs to work with...

This is probably more supervised than you want, but maybe starting with pre-trained weights of this model can get you somewhere faster.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.