In my project I follow the retrieval augmented generation (RAG) approach. I want to create embeddings for my own dataset and use it in combination with llama-2. In the dataset are german annual reports, 548 reports as pdf-files with about 300 sites per report. Next, I want to load the data in a vector store, but first I think I have to create the embeddings.

And now, there are serveral questions and I need some best-practice:

  1. Do I have to train my own embeddings model or can I use models like word2vec of the gensim package or a pretrained model like BERT and take the hidden state?

  2. Can I use any embeddings model? I think they train on a specific corpus and if my words aren't in the training corpus, I will get bad result or what do you think?

  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Aug 28, 2023 at 15:21

1 Answer 1


Depending on the architecture of your RAG system, there can in some cases be efficiencies in using the same embeddings as your LLM of choice, but this is not a hard requirement. For example, if you generate embeddings, store in a vector dB, then take a prompt, embed it, and use it to query your vector dB in order to get a ptr to the original doc, you can use whatever embeddings you see fit as you'll have the original doc to feed back into the LLM.

You'd definitely want to use an embedding model that includes your language of choice, or else multilingual embeddings. I'd recommend testing with multiple different embeddings and seeing what works best for your use case.

  • $\begingroup$ Thank you @brewmaster312 for the answer. Currently I would use Llama-2 to see what is in it with that. My understanding of RAG is that we use the LLM language understanding and do a similarity search, so I agree with you to embed both the data and the prompt. Do I need to take a detailed look at the corpus the embedding model has been trained with, as it is a German financial dataset? $\endgroup$ Commented Aug 28, 2023 at 13:21
  • $\begingroup$ Not sure I understand your question. If you can choose whatever embedding you want, it's also possible to create your own, and this is relatively straightforward. If you're using a pretrained embedding model, I'd recommend one from German or multilingual instead of starting from English alone. I don't know how specific financial jargon is, but you might want to read the FinBERT paper, they got better results from training on financial corpora but they went beyond just embeddings. hth. $\endgroup$ Commented Aug 28, 2023 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.