1
$\begingroup$

I am creating 2 neural machine translation model (model A and B with different improvements each model) with fairseq-py. When I evaluate model with bleu score, model A BLEU score is 25.9 and model C is 25.7. Then i filtered data by length into 4 range values such as 1 to 10 words, 11 to 20 words, 21 to 30 words and 31 to 40 words. I re-evaluated on each filtered data and all bleu scores of model B is greater than model A. Do you think this is normal case?

$\endgroup$
1
$\begingroup$

The original BLEU scores 25.9 and 25.7 are very close, there might not even be any significant difference. It's totally possible that model B performs better than model A on the filtered data only by chance. It's also possible that model B actually performs better than model A on shorter sentences. And finally it's worth noting that BLEU score is based on the number of n-grams in common, so it's likely to be affected by the length of sentences independently from the model being tested.

Conclusion: based on the information provided, this difference seems perfectly reasonable.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.