20

The original paper "BLEU: a Method for Automatic Evaluation of Machine Translation" contains a couple of numbers on this: The BLEU metric ranges from 0 to 1. Few translations will attain a score of 1 unless they are identical to a reference translation. For this reason, even a human translator will not necessarily score 1. It is important to note that ...


11

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder. In Neural Machine Translation (NMT) models, each token of the translation ...


11

The definition of a Language Model (LM) is a probability distribution over sequences of words. The simple illustration of an LM is predicting the next word given the previous word(s). For example, if I have a language model and some initial word(s): I set my initial word to My My model predicts there is a high probability that name appears after My. By ...


7

BLEU (Bi Lingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been machine-translated (MT) from one natural language to another. BLEU is typically measured on a 0 to 1 scale, with 1 as the hypothetical “perfect” translation. Google uses the same definition of BLEU but multiplies the typical score by 100.


5

The basic reasoning, I think, is just to increase capacity. While it is possible in theory for a single head, using multiple simply makes it easier. More specifically though, the paper says (pg 4): Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention ...


5

BLEU scores are based on comparing the translation to evaluate against a gold-standard translation. In general the gold-standard translation is the same source sentence translated by a professional translator, so in theory a professional human translation should always receive the maximum score of 1 (BLEU scores are normalized between 0 and 1). However it's ...


4

(This answer was originally a comment) You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very ...


3

This isn't my area of specialty and I'm not familiar with Moses, but I found this after some searching. I think you are looking for GIZA++. You'll see GIZA++ listed in the "Training" section (left menu) on the Moses home page, as the second step. GIZA++ is briefly described in tutorial fashion here. Here are a few tutorial PowerPoint slides: http://www....


3

Google translate itself uses Deep learning to translate sentences which can be seen here. You can translate sentences across languages for which you need two things : A large dataset which has pairs of translations ( like English-French ). You can find such a dataset from here. A sequence-to-sequence RNN model. They have Encoder-Decoder architecture which ...


3

In encoder-decoder architecture, we first represent the input sequence by a fixed vector. It is assumed that this fix vector represents the complete meaning of the sentence. Now decoder uses this fix vector to generate the output sequence. Answers of your questions: If you use 1 recurrent unit, it outputs the value at each time instant which is not ...


3

A context window applies to the number of words you will use to determine the context of each word. Like of your statement is "the quick brown fox" a context window of two would mean your samples are like (the,quick) and ( the, brown). Then you slide one word and your samples become (quick, the ), (quick, brown) and (quick fox) and so on. I would suggest ...


3

I just googled: A Convolutional Encoder Model for Neural Machine Translation, by Gehring et al., link Convolutional Sequence to Sequence Learning, by Gehring et al. link Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction, by Elbayad et al. link All the implementations I found on GitHub are in PyTorch. I'm not ...


3

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and replace the encoder with XLM-R. Another option would be using Huggingface's Transofrmers that also provides basic support for sequence-to-sequence models as ...


3

First, note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves, so the predictions are not affected. Then, why adding 1 ? Because Tokenizer.word_index is a python dictionary that contains token keys (string) and token ID values (integer), and where the first token ID is 1 (not zero) and where the token IDs are assigned ...


3

At this scale, grep is fast enough, so it's just a matter of having the files in a shape that's convenient for the most common types of searches. At ModelFront, we use and recommend Linear TSV, because TSV has first-class support in bash. https://modelfront.com/docs/eval/#linear-tsv explains the standard and why it's such a natural fit for machine ...


2

I could try to explain you with words, but these slides explain it very well with pictures. Hope it helps. http://www.inf.ed.ac.uk/teaching/courses/mt/lectures/phrase-model.pdf Note this slides correspond to the chapter 5 of "Statistical Machine Translation" by Philipp Koehn, highly recommended if you are working on machine translation, and it is easy to ...


2

I think I have found the answer, but I'd like to have some validation from the community. Could someone please let me know if this seems like a valid explanation? The reason for the remaining 320M parameters in the input of the encoder, computing the word embeddings requires taking a one-hot vector of 160K words and transforming it into a 1K dimensional ...


2

Also the paper said, we do not have a complete explanation to this phenomenon. But here is kinda explanation about that. While we do not have a complete explanation to this phenomenon, we believe that it is caused by the introduction of many short term dependencies to the dataset. Normally, when we concatenate a source sentence with a target ...


2

The most widely used ones in text summarization research is the DUC dataset. If you see a paper using dataset "DUC 2015" or "DUC 2016" that's from here. I have also personally used the Reuters arcihve. You just need to download each article with wget or something similar. See also here. The CNN / DailyMail dataset is also widely used in summarization ...


2

The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the selection of a dictionary. If the word is not in the dictionary we have chosen, then we are saying we have no valid representation for that word (or we are simply not interested). Other tags are ...


2

Here is an implementation from PyPI.


2

There seems to be an issue in Keras save_weights and load_model functions. You can read more about it here, here and here. So far there is no workaround about it.


2

I think your case would benefit from tunning a existing language to the new one but that is only a good approach if you plan to use it commercially. Also Google accepts help to improve their translation algorithm and you could petition for them to assemble a team for this and donate data. Google uses a Neural Network that they call Google's NMT (Neural ...


2

I find this way of using BERT in my translation system and it allows me to load and use more data to train my model. I got a memory error when I want to use more data like 100k for my task. and I came up that my tokenizer is a kind of problem here because it takes a lot of memory to make my tokenizer for such a huge volume of data so pre-trained models like ...


2

BERT is a pretraining model to do the downstream tasks such as question answering, NLI and other language tasks. So it just needs to encode the language representations so that it could be used for other tasks.That's why it consists only of encoder parts. You can add the decoder while doing your specific task and this decoder could be anything based on your ...


2

Rare words are not a problem only for NMT, they are a problem for MT in general. The reason is simple: in order to accurately translate a word in any particular context, the model needs to see as many examples as possible during the training stage. By definition the training data contains very few occurrences of rare words (especially hapax words which occur ...


2

Machine translation using traditional neural architecture (seq2seq models) had various issues due to rare-words, low accuracy and slow translation [1]. Even after using various mechanisms like attention and residual connections the performance was only comparable (not better than) statistical phrase-based machine translations [1]. I can only think of this ...


1

First need to understand what problems BERT can solve or what kind of inference/prediction it can achieve. BERT Neural Network - EXPLAINED! Encoder in Transformer itself can learn: Relations among words (what word is most probable in a context). For instance, what word will fit in the BLANK in the context I take [BLANK] of the opportunity. Relations ...


1

The original BLEU scores 25.9 and 25.7 are very close, there might not even be any significant difference. It's totally possible that model B performs better than model A on the filtered data only by chance. It's also possible that model B actually performs better than model A on shorter sentences. And finally it's worth noting that BLEU score is based on ...


1

In terms of using the data for initial training of a machine learning model, it is a very good idea, because Google Translate produces exceptional results, which will give yo very good ideas about how to go about building and improving your machine learning model and save you time in the long run. As for releasing the dataset to the public, Google has a ...


Only top voted, non community-wiki answers of a minimum length are eligible