Training embeddings on own dataset

Question

In my project I follow the retrieval augmented generation (RAG) approach. I want to create embeddings for my own dataset and use it in combination with llama-2. In the dataset are german annual reports, 548 reports as pdf-files with about 300 sites per report. Next, I want to load the data in a vector store, but first I think I have to create the embeddings.

And now, there are serveral questions and I need some best-practice:

1. Do I have to train my own embeddings model or can I use models like word2vec of the gensim package or a pretrained model like BERT and take the hidden state?

2. Can I use any embeddings model? I think they train on a specific corpus and if my words aren't in the training corpus, I will get bad result or what do you think?

Answer 1

Depending on the architecture of your RAG system, there can in some cases be efficiencies in using the same embeddings as your LLM of choice, but this is not a hard requirement. For example, if you generate embeddings, store in a vector dB, then take a prompt, embed it, and use it to query your vector dB in order to get a ptr to the original doc, you can use whatever embeddings you see fit as you'll have the original doc to feed back into the LLM.

You'd definitely want to use an embedding model that includes your language of choice, or else multilingual embeddings. I'd recommend testing with multiple different embeddings and seeing what works best for your use case.