I want to remove duplicate images from a dataset of 50Million images. What is the best method to detect all the duplicates?

Do you think one shot learning is good for this?

  • $\begingroup$ Exact duplicates? $\endgroup$ – Michael M Sep 28 '18 at 19:16
  • $\begingroup$ No, even augmented ones. $\endgroup$ – thanatoz Sep 28 '18 at 19:17

I think the dhash technique might help. It essentially creates a signature for each image, then you could isolate the duplicated images. 50M could take a while, so perhaps you can try that with a smaller subset and see how well it works.

  • $\begingroup$ Is there a descriptive guide to use this apart from the official jetsetter page? $\endgroup$ – thanatoz Oct 1 '18 at 7:22
  • $\begingroup$ Are you looking for implementation example in a certain language? If you look through the jetsetter article and its references, you can see code sample implementation in C#, PHP, etc. Many people are kind enough to share the code via github as well and hopefully one of them would work for you. $\endgroup$ – The Lyrist Oct 1 '18 at 18:29

So, this is a simple problem that could be solved using one-shot learning technique. To achieve this, we must build a model that understands our data and is capable of finding similarity or dissimilarity in your data.

For this, we must carry out the following steps:

  1. Train (or finetune) the network on dataset of related images.
  2. After training the model, clip the last predicting layers to create embedding.
  3. Pass your testing data through the network and store individual embedding.
  4. Find the difference between the embedding and find the differences crossing a certain threshold.
  5. These images are potentially images having similar data and this could be easily used to find duplicacy in the dataset.


I referred this paper on oneshot learning and later found this blog to be a little helpful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.