How can I apply similarity algorithm (or comparison) of over one million vectors with another one million vectors?

I am following this pyimage search tutorial but don't know how to scale up the algorithm when I have millions of images to compare. Like comparing similarity of over 1 million images with other 1 million images. It stops with an out of memory error at 100000.


You could probably use Dask to do this.

Dask natively scales Python. Dask provides advanced parallelism for analytics, enabling performance at scale for the tools you love

Dask will schedule your computation so that you don't run out of memory and then provide you with the result of your computation. It supports many of the libraries in the data science stack including scikit-learn, pandas and numpy.


You might want to take a look at this project FAISS.

Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM.

It has nice wrappers for you to use from Python. Check the wiki for examples on how you can integrate this in your application.

It is built with scalibilty in mind, supporting indexing and comparisons of millions of vectors and it can be run either on CPU or GPU.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.