BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence

Question

For example I have an original sentence. The word barking corresponds to the word that is missing.

Original Sentence : The dog is barking.
Incomplete Sentence : The dog is ___________.

For example, using the BERT model, it predicts the word crying instead of the word barking. How will I measure the accuracy of the BERT Model in terms of how syntactically correct and semantically coherent the predicted word is?

(For an instance, there are a lot of incomplete sentences, and the task is to evaluate BERT accuracy based on these incomplete sentences.)

In other words, how will I measure the distance in terms of semantics in terms of model between the two words barking and crying.

Please help.

Answer 1

You just stumble over one big problem in the NLP field : finding the perfect metric..

---

Most traditional metrics (BLEU, ROUGE, ...) simply does not take into account the distance in terms of semantics between barking and crying.

So according to these metrics, The dog is crying is as similar as The dog is salmon to the reference, the dog is barking.
From a human viewpoint, this is not correct, the first sentence is closer to the reference, because for example the second sentence makes no sense.

---

People recently tried to provide better metrics in this sense. You might be interested in BERT score.

The idea is simply to use a BERT model (that have been pretrained, therefore have some linguistic knowledge) to compute how similar 2 sentences are.