Neural Network Architecture for Identifying Image Copies

I have a large image collection and wish to identify the images within that collection that appear to copy other images from the collection.

To give you a sense of the kinds of image pairs that I wish to classify as matches, please consider these examples:

I have hand classified roughly .25M pairs of matching images, and now wish to use those hand labelled matches to train a neural network model. I am just not sure which architecture would be ideally suited for this task.

I originally thought a Siamese Network might be appropriate, as they have been used for similar tasks, but the output from those classifiers seems more ideally suited to finding different figurations of the same object (which is not what I want), rather than different printings of the same figuration (which is what I want).

If anyone can help recommend papers or architectures ideally suited to identifying images given the training data I have prepared, I would be tremendously grateful for any insights you can offer.

• "but the output from [Siamese networks] classifiers seems more ideally suited to finding different figurations of the same object" - I actually think Siamese networks would work perfectly. They learn what you tell them to learn. Others have made them learn "different figurations", but you could make them learn image duplication. Dec 18, 2018 at 14:37
• I've just discovered a paper on DEep Local Features (DELF) that combines keypoints based analysis with convolutional neural networks to capture image similarity. Google has created a sample Colab notebook that implements DELF. Jan 31, 2019 at 18:29
• Just a thought, what about variational autoencoders and measure the reconstruction error? I think it would work very well for the left two, but I'm not sure about the right examples. For those, I think a localized approach is required. Sep 19, 2019 at 19:26

You need to read about the triplet loss function. The triplet loss function gets result embeddings from a network, that processes three images by a network (two similar and one non-similar) for one step:

After that the loss is computed as:

$$Loss = \sum\limits_{i=1}^N \big[ \Vert f_i^a - f_i^p \Vert_2^2 - \Vert f_i^a - f_i^n \Vert_2^2 + \alpha \big]_+$$

For more details read the paper from the triplet loss authors.

Also, this may help PSNR, but this is not deep learning.

• The triplet loss is a good suggestion, but link only answers are a bad fit for stack exchange. Can you explain more what the triplet loss is and why it would accomplish what OP wants? Dec 18, 2018 at 14:03
• "Links to external resources are encouraged, but please add context around the link so your fellow users will have some idea what it is and why it’s there. Always quote the most relevant part of an important link, in case the target site is unreachable or goes permanently offline." - datascience.stackexchange.com/help/how-to-answer Dec 18, 2018 at 14:15
• @kbrose I can do copy-paste from a link. Do u think I'm need to do that? Or u ask for TL;DR? Dec 18, 2018 at 14:15
• Removed my downvote. Thank you for adding some context @toodef. Dec 18, 2018 at 14:35
• Sure, I appreciate that point of view. I guess I’m more of the opinion that if you don’t have time then just make a comment. Dec 18, 2018 at 15:19

If the images are more similar like you posted, you can go with Structural similarity index which gives output in the range -1 to 1. any thing more than 0.9 can be considered similar.