How to manage memory constraint and increase speed for 1 vs rest image similarity comparison for over 100k images for computer vision?

Question

I'm looking for ideas on how to do things in a better way, efficiently when using Machine/Deep Learning.

I am working on a search improvement problem using Computer vision where I am thinking about comparing two images and see their similarity score so theoretically there are lots of things I can try such as: Structural Similarity, Template Matching, RMSE, PSNR etc... On the second thought, I can use Siamese Network. So in both cases, I have to search every image with every other image and get top-k scores which shows that k most similar images.

PROBLEM: I have a huge and by huge I mean around 20 Million+ images data. So in theory, let's suppose even if I use 100K images, it'll take huge amount of memory for nXm dimensional image matrix and on top of that, calculating scores with every other image for each image can't work on incoming live data as it'll take too much time. Impossible for Siamese Network as we have to insert all the images one by one.

Please tell me if there is a way to achieve this? I am using MongoDB. How can this be achieved? Someone said that ElasticSearch has a functionality to store image vectors. I have no clue what Elastic is and even if I read it, I don't know how to use that specially with Python?

Can someone suggest how it is achieved in IT industries and big companies because they have huge data and there must be a simpler way to do this.

Answer 1

One option is locality-sensitive hashing (LSH) which hashes similar input items into the same "buckets" with high probability. LSH is frequently used for near-duplicate image detection, a variation of your problem.