# BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence

For example I have an original sentence. The word barking corresponds to the word that is missing.

Original Sentence : The dog is barking.
Incomplete Sentence : The dog is ___________.


For example, using the BERT model, it predicts the word crying instead of the word barking. How will I measure the accuracy of the BERT Model in terms of how syntactically correct and semantically coherent the predicted word is?

(For an instance, there are a lot of incomplete sentences, and the task is to evaluate BERT accuracy based on these incomplete sentences.)

In other words, how will I measure the distance in terms of semantics in terms of model between the two words barking and crying.

Most traditional metrics (BLEU, ROUGE, ...) simply does not take into account the distance in terms of semantics between barking and crying.
So according to these metrics, The dog is crying is as similar as The dog is salmon to the reference, the dog is barking.