4

(This answer was originally a comment) You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very ...


3

At this scale, grep is fast enough, so it's just a matter of having the files in a shape that's convenient for the most common types of searches. At ModelFront, we use and recommend Linear TSV, because TSV has first-class support in bash. https://modelfront.com/docs/eval/#linear-tsv explains the standard and why it's such a natural fit for machine ...


3

First, note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves, so the predictions are not affected. Then, why adding 1 ? Because Tokenizer.word_index is a python dictionary that contains token keys (string) and token ID values (integer), and where the first token ID is 1 (not zero) and where the token IDs are assigned ...


2

Data structures for multi-lingual can be bit tiresome & repetitive, especially if data is not structured properly. Assuming the content of data in en is the same across in other languages en-es, en-fr, en-de. In other words, train.en is same in en-es, en-fr and en-de considering word hello: English(en) its Hello Spanish(es) its Hola French(fr) its ...


1

File systems are good enough for data storage up to a point. As data becomes more complex, it often makes sense to move to a database. A relationship database will allow for database normalization, storing a single, canonical value for each entity. This will greatly reduce the size of the data and allow for different combinations of train, dev, and test ...


1

It encodes the language and its regional variant, the same way as locales are encoded. hi_IN then means Hindi as spoken in India, en_US would mean American English, en_GB British English. My guess is that en_XX means English in general. Anyway, the first part of the locale code is the ISO 639-1 language code which is the same as langid uses. Btw. langid ...


1

As you certainly know, Machine Translation (MT) is a very challenging and useful task in the domain of Natural Language Processing (NLP). As such it is a very specialized research domain but also a very active area of research, and a very competitive one (in particular due to commercial applications, obviously). So there's a massive amount of research being ...


Only top voted, non community-wiki answers of a minimum length are eligible