0
$\begingroup$

I want to calculate the BLEU_SCORE but it gives me bad results I don't know why?

For example, this is my reference and predicted sentences:

ref : 
['electron', 'and', 'a', 'proton']
predicted : 
['electron', 'and', 'a', 'proton']
ref : 
['to', 'reach', 'the', 'nectar', 'at', 'the', 'bottom', 'of', 'flowers']
predicted : 
['to', 'reach', 'the', 'nectar', 'at', 'the', 'bottom', 'of', 'flowers']
ref : 
['during', 'the', 'summer', 'near', 'the', 'north', 'pole']
predicted : 
['during', 'the', 'summer', 'near', 'the', 'north', 'pole']
ref : 
['only', 'blue', 'light', 'is', 'reflected', 'by', 'the', 'block']
predicted : 
['only', 'blue', 'light', 'is', 'reflected', 'by', 'the', 'block']
ref : 
['between', '20', 'and', '40', 'degrees', 'latitude']
predicted : 
['between', '20', 'and', '40', 'degrees', 'latitude']
ref : 
['external', 'and', 'internal', 'combustion', 'engines']
predicted : 
['external', 'and', 'internal', 'combustion', 'engines']
ref : 
['cleaning', 'disinfecting', 'and', 'in', 'swimming', 'pools']
predicted : 
['cleaning', 'disinfecting', 'and', 'in', 'swimming', 'pools']
ref : 
['body', 'mass', 'index', 'bmi']
predicted : 
['body', 'mass', 'index', 'bmi']
ref : 
['they', 'put', 'nutrients', 'into', 'the', 'soil', 'that', 'plants', 'use', 'to', 'grow']
predicted : 
['they', 'put', 'nutrients', 'into', 'the', 'soil', 'that', 'plants', 'use', 'to', 'grow']
ref : 
['structure', 'of', 'earth', 'interior']
predicted : 
['structure', 'of', 'earth', 'interior']

And here is the code I used to calculate the BLEU_SCORE :

from nltk.translate.bleu_score import corpus_bleu


print("Individual n-gram")
print("Individual 1-gram")
print('BLEU-1: %f' % corpus_bleu(ref, pre, weights=(1.0, 0, 0, 0)))
print("Individual 2-gram")
print('BLEU-2: %f' % corpus_bleu(ref, pre, weights=(0, 1.0, 0, 0)))
print("Individual 3-gram")
print('BLEU-3: %f' % corpus_bleu(ref, pre, weights=(0, 0, 1.0, 0)))
print("Individual 4-gram")
print('BLEU-4: %f' % corpus_bleu(ref, pre, weights=(0, 0, 0, 1.0)))

OUTPUT:

Individual n-gram
Individual 1-gram
BLEU-1: 0.015625
Individual 2-gram
BLEU-2: 0.000000
Individual 3-gram
BLEU-3: 0.000000
Individual 4-gram
BLEU-4: 0.000000

Anyone can help with this I don't know why it doesn't give me good results?

$\endgroup$
1
$\begingroup$

You must be getting a warning with this output. Warning pretty much tells you the reason why the scores are 0. Because there ARE NO 2-gram , 3 -grams in you example which are overlapping.

enter image description here

Here is the detailed explanation, I couldn't explain it better- https://github.com/nltk/nltk/issues/1838

EDIT- Solution-

Although the warning tells the reason, here is how you can fix this-

Notice the ref and pre,

from nltk.translate.bleu_score import corpus_bleu
ref =[[['electron', 'and', 'a', 'proton']]]
pre =[['electron', 'and', 'a', 'proton']]


print("Individual n-gram")
print("Individual 1-gram")
print('BLEU-1: %f' % corpus_bleu(ref, pre, weights=(1.0, 0, 0, 0)))
print("Individual 2-gram")
print('BLEU-2: %f' % corpus_bleu(ref, pre, weights=(0, 1.0, 0, 0)))
print("Individual 3-gram")
print('BLEU-3: %f' % corpus_bleu(ref, pre, weights=(0, 0, 1.0, 0)))
print("Individual 4-gram")
print('BLEU-4: %f' % corpus_bleu(ref, pre, weights=(0, 0, 0, 1.0)))

enter image description here

You can refer help of python-

enter image description here

$\endgroup$
7
  • $\begingroup$ But in my example there are 2-grams,3-grams and also there are 4-grams too? $\endgroup$ May 20 '20 at 18:10
  • $\begingroup$ stackoverflow.com/questions/40542523/… . This is a better explanation. $\endgroup$ May 20 '20 at 18:41
  • 1
    $\begingroup$ It help me understand more how the BLEU work but didn't fixe my problem, do you have any ideas about the smoothing_function? $\endgroup$ May 20 '20 at 22:20
  • $\begingroup$ I have updated the answer with the fix.and also gave another link to explanation. $\endgroup$ May 20 '20 at 22:22
  • 1
    $\begingroup$ I see the difference now! the difference that the references should be a list(list(list(string))) and the predicted should be just a list(list(string)), Thanks! $\endgroup$ May 20 '20 at 22:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.