How to evaluate Natural Question-Answer Generation pairs?

Question

I am trying to generate Natural Question-Answer for a specific domain. I am using a Large Language Model (LLM). I have only context to generate question-answers but don't have any ground truth. How to measure the accuracy or how good the generation is? I am repeating the experiment 2-3 times, How to compare which question-answers pairs are good? Because each time the generated question-answers are different.

For example,e :

Context :

"""This section describes our proposed method. The detailed setup for our experiments is described in Sections 4.1 and 4.2."""

Iteration1 (generated question-answer)

Q: This section describes what?
A: This section describes the paper's proposed model.

Iteration2 (generated question-answer)

Q: Which section describes the detailed experiments?
A: Sections 4.1 and 4.2 describes the detailed setup and experiments.

Iteration2 (generated question-answer)

Q: Sections 4.1 and 4.2 describes what?
A: Detailed setup and experiments are described in Sections 4.1 and 4.2

Now I want to measure how good this model is in generating questions and answers based on the given paragraph. What matrices I can use? Please guide me on this, and if possible share the papers too.

Answer 1

I assume you are using a seq2seq transformer like T5. And the input -> output is probably defined as <context> -> question: ..., answer: ....

Since you do not have any ground truth, I would use another model, a Gold Standard/SOTA if you may, in Question Answering, where its input -> output is question:... , context:... -> <answer>. Hence, I would use the answers of your LLM and evaluate it against the Gold standard where I would use a BLEU score (or something else for text generation) between the answer of your LLM and the answer of the QA Gold Standard.

Of course, in the limitations you should acknowledge the fact that this evaluation is not ideal, since you do not have ground truth. Or maybe you could try using a few QA models and average. You should be careful with the hyper-parameters. Imagine one's sequence output length is less than the other.

Answer 2

Accuracy can not be measured without ground truth, the generated answers have to be labeled. The labels can be generated by a human or another established LLM.