Evaluation of a wide variety of natural language generation (NLG) tasks is difficult. For instance, for a question answering model, it is hard for a human to quantify how well the model has answered a particular question. Doing this at scale is even harder, because it requires automating that judgement about output quality.
The most common approach for evaluation of NLG at scale involves building a set of test queries and reference answers, where the reference answers set out the 'gold standard' for how the model should respond. In the case of a Q&A bot, this would be a list of questions and 'good' answers; for a machine translation system, this would be some human-verified translations.
A good text generation model ideally takes a query as input and returns an output as close as possible to the reference given in the test set. As such, the model is assessed by passing in each query in turn, and comparing how semantically similar the model's output is to the reference output. If the model's output is similar to the reference, then this means the model is performing well.
My question is how do we assess semantic similarity between the reference and candidate answers?
A few ideas:
- Old-school string matching - Calculate word, subword, or n-gram overlap between candidate and reference answers. Can use metrics like F1-score or recall depending on use case. The idea is that a good answers includes as much of the surface content from the reference as possible, with as little extraneous information as possible. However, this sort of approach performs poorly where the meaning of the answer is the same, but the surface form is different - or vice versa, e.g. 'The cat is under the mat' and 'The mat is under the cat' have a different meaning but contain all the same unigrams, so would get a high similarity score with a string-based metric.
- Vector distance between embeddings - Use text embeddings trained to map paraphrases to similar embeddings. Encode the reference and candidate answers, and then use a measure of vector distance (e.g. cosine-similarity) to evaluate. If - once encoded - the reference and candidate answers are 'close' then the two answers should be a near-paraphrase of one another. This means the candidate answer does a good job of including the meaning from the reference answer. However, this method is only as good as the embeddings underpinning it. Moreover, it seems circular to use semantic similarity to evaluate the outputs of tasks where semantic similarity is used to produce the outputs (as is typically the case for Q&A bots, semantic search, summarisation, machine translation, etc.)
- Mover distance in semantic space* - This is a similar approach to using vector distance. The idea is that encoded text can be visualised as $n$ points in $k$-dimensional space, where $n$ is the number of tokens in the text and $k$ is the dimension of the embeddings used. Then, we can think about the candidate and reference answers being $n_c$ and $n_r$ points in that semantic space. We can then think about moving candidate points to sit on top of reference points - the total distance involved in this movement is the 'mover distance'. There exists some optimal, i.e. most efficient, way of moving candidate points to sit on top of reference points, and this gives the 'mover score' for that model on that query.
What approaches have I missed? What are the strengths and weaknesses of each approach? Are there some state-of-the-art approaches that outperform these?
