I'm trying to implement Text Summarization task using different algorithms and libraries. To evaluate which one gave the best result I need some metrics. I have read about the Bleu and Rouge metrics but as I have understand both of them need the human reference summaries as a reference. Is it so that no automatic evaluation can be done in text summarization, i.e human generated summary is needed beforehand? What is the common practice for the tasks like this? How often is the text summarization accuracy measured at all?

  • $\begingroup$ Without reference summaries, you cannot get an automatic metric for assessing the quality of the summary. However there is other measure, such as n-gram novelty, for measuring the abstractiveness of a summary for example. It does not need reference summaries. $\endgroup$ – Astariul Jun 1 at 7:19

So, your question is talking about whether human reference summaries are required to evaluate summarisation models.

The short answer is yes at the moment. The most important things about an output summary that we need to assess are the following:

  1. The fluency of the output text itself (related to the language model aspect of a summarisation model)
  2. The coherence of the summary and how it reflects the longer input text.

The problem with have an automatic evaluation system for a text summarisation model is that, although we can assess fluency from a language model, we can't really assess whether the model has pulled "the most salient" pieces of information from the original, longer text (and this can subjective from person to person). Hence why we need multiple human reference summaries to compute ROUGE and BLEU. However, as you are aware, these metrics have their limitations.

ROUGE is essentially a further development of BLEU, which have been commonly used a dubious proxy for output text fluency in research to compare summarisation and translation models. These metrics are dubious because they simply look at how much they overlap with reference texts from humans (https://rxnlp.com/how-rouge-works-for-evaluation-of-summarization-tasks/#.Xt1ewy-ZOi4).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.