Evaluation of a wide variety of natural language generation (NLG) tasks is difficult. For instance, for a question answering model, it is hard for a human to quantify how well the model has answered a particular question. Doing this at scale is even harder, because it requires automating that judgement about output quality.

The most common approach for evaluation of NLG at scale involves building a set of test queries and reference answers, where the reference answers set out the 'gold standard' for how the model should respond. In the case of a Q&A bot, this would be a list of questions and 'good' answers; for a machine translation system, this would be some human-verified translations.

A good text generation model ideally takes a query as input and returns an output as close as possible to the reference given in the test set. As such, the model is assessed by passing in each query in turn, and comparing how semantically similar the model's output is to the reference output. If the model's output is similar to the reference, then this means the model is performing well.

My question is how do we assess semantic similarity between the reference and candidate answers?

A few ideas:

  • Old-school string matching - Calculate word, subword, or n-gram overlap between candidate and reference answers. Can use metrics like F1-score or recall depending on use case. The idea is that a good answers includes as much of the surface content from the reference as possible, with as little extraneous information as possible. However, this sort of approach performs poorly where the meaning of the answer is the same, but the surface form is different - or vice versa, e.g. 'The cat is under the mat' and 'The mat is under the cat' have a different meaning but contain all the same unigrams, so would get a high similarity score with a string-based metric.
  • Vector distance between embeddings - Use text embeddings trained to map paraphrases to similar embeddings. Encode the reference and candidate answers, and then use a measure of vector distance (e.g. cosine-similarity) to evaluate. If - once encoded - the reference and candidate answers are 'close' then the two answers should be a near-paraphrase of one another. This means the candidate answer does a good job of including the meaning from the reference answer. However, this method is only as good as the embeddings underpinning it. Moreover, it seems circular to use semantic similarity to evaluate the outputs of tasks where semantic similarity is used to produce the outputs (as is typically the case for Q&A bots, semantic search, summarisation, machine translation, etc.)
  • Mover distance in semantic space* - This is a similar approach to using vector distance. The idea is that encoded text can be visualised as $n$ points in $k$-dimensional space, where $n$ is the number of tokens in the text and $k$ is the dimension of the embeddings used. Then, we can think about the candidate and reference answers being $n_c$ and $n_r$ points in that semantic space. We can then think about moving candidate points to sit on top of reference points - the total distance involved in this movement is the 'mover distance'. There exists some optimal, i.e. most efficient, way of moving candidate points to sit on top of reference points, and this gives the 'mover score' for that model on that query.

What approaches have I missed? What are the strengths and weaknesses of each approach? Are there some state-of-the-art approaches that outperform these?

*Colombo, et al. (2021), Zhao, et al. (2019)


1 Answer 1


Well, you missed the old good human evaluation, which is the only actual measure that can actually be trusted in terms of semantic evaluation. Also, in the reference n-gram matching area you missed BLEU, which is the standard evaluation measure in machine translation, and METEOR, which applies stemming and handles synonyms to avoid the surface word problem you mentioned.

Apart from that, in the space of embedded vector approaches, it is worth mentioning BERTscore, which seems to work very very well. This is the abstract of the article, published at ICLR'20:

We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.

These are some references to better understand BERTscore's measures and their traits:

Also worth mentioning the WMT Metrics Shared Task, which aims at devising automatic evaluation metrics for machine translation. There, each year you can find practical comparisons of the submitted evaluation metrics.

  • $\begingroup$ Thanks for this. BERTScore is really interesting, from my understanding it is really a development on the vector distance idea - but rather than computing the distance between encodings of whole sentences/documents, you compute token-level distances and then aggregate up by taking the maximum similarity for each token (in this way it approximates the mover distance approach). What do you think of concerns about circularity? In question answering, for instance, we return text chunks from a database according to encoded vector distance - can we still use these types of scores to evaluate? $\endgroup$
    – Greggs
    Feb 23 at 12:27
  • 1
    $\begingroup$ Sorry, I'm not too familiar with QA and the "circularity" issue you mention. Nevertheless, BERTscore has been recently evaluated for QA: Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics (ACL Findings'22) $\endgroup$
    – noe
    Feb 23 at 13:08
  • $\begingroup$ Looking at the correlation between BERTScore/LERC and human judgements from that paper (and similar results reported here for a wider set of tasks, The Glass Ceiling of Automatic Evaluation in Natural Language Generation) it does seem like - as you say - there is no good substitute for human evaluation... $\endgroup$
    – Greggs
    Feb 23 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.