3
$\begingroup$

I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the application again:

  • text corpora (few hundred thousand documents, containing a few paragraphs)
  • embeddings create with BERT (for each paragraph)
  • Application: similarity search (retrieve similar paragraphs and reference to the document)

I've seen tutorials about creating embeddings with BERT etc. and it all works. The Crux I have is how to manage having a few million embeddings and searching for similar ones. Where to store them, plus the additonal information (raw text related to the embeddings and document which contains the text).
So the question is:
How does one store a few million embeddings (768-Dimensional numpy arrays) in an efficient and searchable way without using cloud-environments (data privacy reasons)?
Is Tensorflow Records the right answer?
Is it in the end a relational database?
Is it something different? It's my first NLP task and I might simply not know the obvious answer. However, searching on stackexchange and google didn't provide a solution.

Improve this question
$\endgroup$
8
  • $\begingroup$ I'm not aware of any standard approach for this kind of use case. In academia at least the NLP community tends not to bother with databases... but I can imagine that it's a different story in a production system. $\endgroup$
    – Erwan
    Nov 28, 2019 at 2:38
  • $\begingroup$ @Erwan Yes, this is where my problem comes from. There are plenty tutorials for using bert to get embeddings, but nobody tells me how to handle 4 Million of them efficiently. I have to add: I said it's a production environment, but it still is a research project in academia. It's about comparing performant systems. However, I need a reasonable approach to handling embeddings before talking about performance ;) $\endgroup$
    – Angus
    Nov 28, 2019 at 10:14
  • $\begingroup$ Raw text IS your database. Raw text contains all the information of the text, in a small format. Storing token embeddings present little interest. What's your goal by doing this ? $\endgroup$
    – Astariul
    Nov 29, 2019 at 0:46
  • $\begingroup$ @Astariul my goal is doing similarity search via embeddings. So I need to know the embeddings of all text units in my database. Of course I need to store them. I can't calculate 4 Million embeddings every time I want to search for a similar item. $\endgroup$
    – Angus
    Dec 2, 2019 at 13:39
  • $\begingroup$ @Angus can I check how long it takes for you to calculate 4 million embeddings? just curious $\endgroup$
    – lppier
    Dec 12, 2019 at 6:02

3 Answers 3

Reset to default
2
$\begingroup$

There's Milvus search engine that utilizes several prominent Approximate KNN libraries such as FAISS, ANNOY and HNSW. It also handles several bookkeeping, clustering, data integrity and other tasks that you probably don't want to handle yourself. All for a performance price ofc, but if you don't want to pay it, you can always pick one of the "barebones" libraries.

Improve this answer
$\endgroup$
0
$\begingroup$

Why don't you cluster similar embeddings and store then use hashing to search faster. Then you can store them anywhere, maybe in bigdata hdfs distributed system for faster retrieval or simply hashed clusters in databases if you are in research or POC environment.

I have also seen some other techniques for information retrieval, in which you apply TF IDF or simpler search techniques for first filtering out the text of interest and then work on 768 dim embeddings. This way is faster if search is your primary target.

Improve this answer
$\endgroup$
0
$\begingroup$

My answer would be it depends on your creativity. I've seen people storying them in numpy files, pickle files, graph databases and etc.

So I would say it doesn't matter where you store them, It's your code that needs to adapt to the stored files.

For similarity search, you can use indexing algorithms to make it faster. FAISS is a solution to this.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.