I am working on a model to build questions automatically from some text

My model will analyse provided article and ask authors questions that can help improving their articles

How can we measure the accuracy of these ML-generated questions?

There is the relevance part of the questions as these questions represent an area of improvement in the article

How to measure that?

Any previous work on similar models would be a great help too



1 Answer 1


You can check the Question Generation section of paperswithcode. There, you can see for different datasets how the performance is measured and how different proposed approaches compare on them.

Usually, you check how similar is the question to the reference text. Some used measures are BLEU-1 (based on matching unigrams) and ROUGE-L (based on the longest common subsequence). This is "unsupervised testing" in the sense that you don't need labeled data. However, they may not be directly correlated with their actual quality (see Towards a better metric for evaluating question generation systems and Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering).

In other cases, they use QA-based Evaluation (QAE), which measures how similar the generated QA pairs are compared to some ground truth QA pairs. For this, you need a reference labeled QA dataset on which the model is to be evaluated.

  • $\begingroup$ If you find the answers useful, please consider upvoting it and accepting it if you consider it correct or, alternatively, please describe in a comment why you consider it incorrect or not clear enough. $\endgroup$
    – noe
    Commented Jul 29, 2023 at 18:13
  • $\begingroup$ Thanks, but these measures focus on the "Answerability" of the questions.. What I am looking for is to measure the "Relativity" of the question; as is this the right question to ask? $\endgroup$
    – asmgx
    Commented Jul 30, 2023 at 15:40
  • $\begingroup$ QAE focuses on answerability, but the others (BLEU, ROUGE) do not; you can check Towards a Better Metric for Evaluating Question Generation Systems) regarding this. How do you define "relativity"? Are you referring to how related those questions are to the reference text? $\endgroup$
    – noe
    Commented Jul 30, 2023 at 20:32
  • $\begingroup$ yes i meant how relavent those questions are to the reference text $\endgroup$
    – asmgx
    Commented Jul 30, 2023 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.