Fine-tuning LLM with limited documents and hierarchy

Question

Hello LLM enthusiasts.

I am wondering w.r.t. a neighbouring project if there are state of the art approaches to fine tune a model if:

1. the realm of documents is limited (still more than just a few),
2. these documents are regularly in a relationship.

Actually all legal people should have this challenge with 2., too. Shouldn't fine tuning be a lot more efficient if one had sth like a document entity relationship model to add granularity and weights from the very start?

In the end for all applications that rely on a minimum of hallucination it should be important to get as much accuracy as possible to be able or allowed to use the model at all.

(As mentioned this should be a thing for all legal staff but also medical ppl.)

Thank you for your ideas, feel free to course correct me w/o mincing words.

Or am I just using wrong words here and what I mean is "simply" a knowledge graph?

Answer 1

Well I can answer it now.

What I was looking for is RAG - Retrieval-augmented generation.

Actually I wonder how far hierarchies get be respected there, but in general, yes, it is a RAG.

:) case closed.