Why does BLEU score for ignite, torchmetrics and nltk differs?

Question

Here is the example :

from ignite.metrics.nlp import Bleu
from nltk.translate.bleu_score import sentence_bleu
from torchmetrics.text.bleu import BLEUScore


references = [['the', 'quick', 'brown', 'fox', 'jumped', 'over', 'the', 'lazy', 'dog']]
candidate = ['the', 'quick', 'brown', 'fox', 'jumped', 'over', 'the', 'lazy', 'dog','and','the','cat']


#using nltk
score = sentence_bleu(references, candidate)
print(score)

#using torch_metrics
bleu = BLEUScore()
print(float(bleu(candidate,reference)))

#using ignite

bleu = Bleu()
bleu.reset()
bleu.update((candidate,references))
print(float(bleu.compute()))

# 0.7102992180127422
# 0.0
# 0.0

with the tested version :

import ignite
print(ignite.__version__)
import torchmetrics
print(torchmetrics.__version__)
import nltk
print(nltk.__version__)

#0.4.11
#0.11.4
#3.8.1

What am I missing? The dynamic of values on nltk seems better than those of the torchmetrics and ignite frameworks ? can we obtain similar values, with a tweak of the respective parameters of each function ? Thank you for your time.

Answer 1

Torch metrics expects untokenized sentences:

#using torch_metrics
references = [['the quick brown fox jumped over the lazy dog']]
candidate = ['the quick brown fox jumped over the lazy dog and the cat']
torchmetrics_bleu = BLEUScore()
print(float(torchmetrics_bleu(candidate, references)))
# 0.7102992534637451

For ignite you are not providing the expected types of inputs, because update expect one more list nesting level:

# using ignite
ignite_bleu = Bleu()
ignite_bleu.reset()
ignite_bleu.update(([candidate], [references]))
print(float(ignite_bleu.compute()))
# 0.7102992255729442