I need to train a system on a large set of images and associated captions to determine which (image, caption) pairs are correct and which are not. I don't have any labeled pairs, but I can assume that more pairs are correct than are not. Furthermore, the images and captions have small subsets that are reasonably similar, but not exactly.
My current idea is to build a sort of GAN architecture where I have a generator and a discriminator, except that the generator is simply sampling from the existing corpus of images and captions to generate false pairs and the discriminator is in charge of determining which pairs are correct or not.
images, captions = ,  bpg = bad_pair_generator(images, captions) clfr = classifier() for epoch in range(epochs): good_pairs, bad_pairs = zip(images, captions), bpg.generate(N) X, y = good_pairs + bad_pairs, *len(good_pairs) + *len(bad_pairs) clfr.train(X, y) trickiness = [yi == yi_pred for yi, yi_pred in zip(y, clfr.predict(X))] bpg.fit(bad_pairs, trickiness[-len(bad_pairs):])
After this, I would hope that the classifier was stressed sufficiently to learn a good model for whether the pairs are good or not.
- Are there better researched approaches?
- If not, does this approach seem sane and reasonably safe?
- What kind of model could I use to compared the image to the text? Clearly a CNN/RNN would be good for the image/text, but how do you combine them?