High Level Understanding of Neural Machine Translation

Question

I am currently studying Deep Learning based Machine Translation systems but I'm not sure in my understanding the logic of the process. I understand that the source and destination language translation sentence pairs must be represented as word2vec vectors, but why is it necessary to apply two (encoder-decoder) Recurrent Neural Networks? My first idea would be applying only one RNN, where the input is the source language examples (in the form of word2vec vectors) and the output is simply a word2vec sequence of the destination language. Why is it necessary to use another RNN?

My additinal question is if this system is flexible enough to cope with synonimes, word order variations and other disambiguities? Is it capable of approximate the correct meaning of a new to-be-translated source language sentence?

And last but not least: how could one evaluate such a model where many translations can be correct at the same time?

Answer 1

In encoder-decoder architecture, we first represent the input sequence by a fixed vector. It is assumed that this fix vector represents the complete meaning of the sentence. Now decoder uses this fix vector to generate the output sequence.

Answers of your questions:

1. If you use 1 recurrent unit, it outputs the value at each time instant which is not suitable for machine translation as translation of first word may be in the last or somewhere else in target language.(I am not sure how will you map the output and input in that case.)
2. Regarding synonyms, system can handle synonyms as words which are synonyms have very close word vectors.
3. Regarding Word-Order, it can learn the word order if you have enough training data. Decoder also uses beam search to find the best possible translation. I will suggest you to read this paper thoroughly. https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf
4. Regarding evaluation, there are various metrics which are designed to evaluate the machine translation system. BLEU is one of the most famous one out of all. It uses various reference translations. You can read more about BLEU in the paper by Kishore Papineni et. al.(I can not post more than two link in answer as my reputation is less than 10.)

I will also suggest you to read the below paper regarding neural machine translation. https://arxiv.org/pdf/1409.0473.pdf

Hope it helps!!