i've read some papers about machine translation. Authors usually define a threshold to limit vocabulary to minimum rare words and replace rare words into token. So the question is if we increase the value of threshold, does it make the translation result (BLEU) improve ? Or it might decrease ? Because i'm creating some nmt models with different improvements, i've tested on 3 number of threshold (0, 2 and 5), BLEU score of base model (without improvement) is better than others when use threshold 5 but with threshold 0 and 2, my improved models is better. Is this normal behavior ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.