I using the HuggingFace library to do sentence paraphrasing (given an input sentence, the model outputs a paraphrase). How am I supposed to compare the results of two separate models (one trained with t5-base, the other with t5-small) for this task? Can I just compare the validation loss or do I need to use a metric (if so, what metric)?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Evaluation should always be specific to the target task and preferably rely on some unseen test set.
The target task is paraphrasing, so the evaluation should be designed to check externally how good the generated sentences are as paraphrases. Usually this kind of task (the output is similar to Machine Translation) is evaluated by using a gold standard set of paraphrases and measuring how close the generated paraphrase is to the gold standard paraphrase(s). The comparison commonly uses some variant of BLEU score.
Practically the way to know how to evaluate a common task is to search the state of the art: for example this paper uses the method described above with various similarity measures (BLEU, ROUGE and variants apparently).