3

First, note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves, so the predictions are not affected. Then, why adding 1 ? Because Tokenizer.word_index is a python dictionary that contains token keys (string) and token ID values (integer), and where the first token ID is 1 (not zero) and where the token IDs are assigned ...


1

It encodes the language and its regional variant, the same way as locales are encoded. hi_IN then means Hindi as spoken in India, en_US would mean American English, en_GB British English. My guess is that en_XX means English in general. Anyway, the first part of the locale code is the ISO 639-1 language code which is the same as langid uses. Btw. langid ...


1

As you certainly know, Machine Translation (MT) is a very challenging and useful task in the domain of Natural Language Processing (NLP). As such it is a very specialized research domain but also a very active area of research, and a very competitive one (in particular due to commercial applications, obviously). So there's a massive amount of research being ...


1

According to recent publications, it is not impossible to get BLEU scores as high as yours for English→Irish. Nevertheless, without any other knowledge, they certainly seem too high. From the command line arguments, there does not seem to be any evident problem. The most probable explanation is, as you already pointed out, a data leakage between validation/...


1

(This answer was originally a comment) You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very ...


Only top voted, non community-wiki answers of a minimum length are eligible