I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the application again:
- text corpora (few hundred thousand documents, containing a few paragraphs)
- embeddings create with BERT (for each paragraph)
- Application: similarity search (retrieve similar paragraphs and reference to the document)
I've seen tutorials about creating embeddings with BERT etc. and it all works. The Crux I have is how to manage having a few million embeddings and searching for similar ones. Where to store them, plus the additonal information (raw text related to the embeddings and document which contains the text).
So the question is:
How does one store a few million embeddings (768-Dimensional numpy arrays) in an efficient and searchable way without using cloud-environments (data privacy reasons)?
Is Tensorflow Records the right answer?
Is it in the end a relational database?
Is it something different? It's my first NLP task and I might simply not know the obvious answer. However, searching on stackexchange and google didn't provide a solution.