I'm looking for ideas on how to do things in a better way, efficiently when using Machine/Deep Learning.
I am working on a search improvement problem using Computer vision where I am thinking about comparing two images and see their similarity score so theoretically there are lots of things I can try such as:
Structural Similarity, Template Matching, RMSE, PSNR etc... On the second thought, I can use
Siamese Network. So in both cases, I have to search every image with every other image and get top-k scores which shows that k most similar images.
PROBLEM: I have a huge and by huge I mean around 20 Million+ images data. So in theory, let's suppose even if I use 100K images, it'll take huge amount of memory for
nXm dimensional image matrix and on top of that, calculating scores with every other image for each image can't work on incoming live data as it'll take too much time. Impossible for
Siamese Network as we have to insert all the images one by one.
Please tell me if there is a way to achieve this? I am using
MongoDB. How can this be achieved? Someone said that
ElasticSearch has a functionality to store image vectors. I have no clue what Elastic is and even if I read it, I don't know how to use that specially with Python?
Can someone suggest how it is achieved in IT industries and big companies because they have huge data and there must be a simpler way to do this.