1
$\begingroup$

Here is the example :


from ignite.metrics.nlp import Bleu
from nltk.translate.bleu_score import sentence_bleu
from torchmetrics.text.bleu import BLEUScore


references = [['the', 'quick', 'brown', 'fox', 'jumped', 'over', 'the', 'lazy', 'dog']]
candidate = ['the', 'quick', 'brown', 'fox', 'jumped', 'over', 'the', 'lazy', 'dog','and','the','cat']


#using nltk
score = sentence_bleu(references, candidate)
print(score)

#using torch_metrics
bleu = BLEUScore()
print(float(bleu(candidate,reference)))

#using ignite

bleu = Bleu()
bleu.reset()
bleu.update((candidate,references))
print(float(bleu.compute()))

# 0.7102992180127422
# 0.0
# 0.0

with the tested version :

import ignite
print(ignite.__version__)
import torchmetrics
print(torchmetrics.__version__)
import nltk
print(nltk.__version__)

#0.4.11
#0.11.4
#3.8.1

What am I missing? The dynamic of values on nltk seems better than those of the torchmetrics and ignite frameworks ? can we obtain similar values, with a tweak of the respective parameters of each function ? Thank you for your time.

$\endgroup$

1 Answer 1

0
$\begingroup$

Torch metrics expects untokenized sentences:

#using torch_metrics
references = [['the quick brown fox jumped over the lazy dog']]
candidate = ['the quick brown fox jumped over the lazy dog and the cat']
torchmetrics_bleu = BLEUScore()
print(float(torchmetrics_bleu(candidate, references)))
# 0.7102992534637451

For ignite you are not providing the expected types of inputs, because update expect one more list nesting level:

# using ignite
ignite_bleu = Bleu()
ignite_bleu.reset()
ignite_bleu.update(([candidate], [references]))
print(float(ignite_bleu.compute()))
# 0.7102992255729442
$\endgroup$
4
  • $\begingroup$ Thanks Sir : ), indeed I supposed the api would be the same! I will let this post here, in case others make the same mistake. Have a good day . $\endgroup$ Mar 15, 2023 at 15:33
  • $\begingroup$ Please, consider upvoting and accepting the answer. $\endgroup$
    – noe
    Mar 22, 2023 at 9:53
  • $\begingroup$ I tried but can't cast a vote, I have a new account.. $\endgroup$ Mar 22, 2023 at 10:21
  • $\begingroup$ It's Ok to just accept the answer with the tick (✓) next to it $\endgroup$
    – noe
    Mar 22, 2023 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.