0
$\begingroup$

For transformer-based neural machine translation (NMT), take English-Chinese for example, we pass English for encoder and use decoder input(Chinese) attend to encoder output, then final output.

What if we do not pass input for decoder and consider it as a 'memory' model for translation. Is it possible and what will happen?

It seems decoder could be removed and there only exist encoder.

Could I do translation task like text generation? See:

https://github.com/salesforce/ctrl/blob/master/generation.py

https://einstein.ai/presentations/ctrl.pdf

$\endgroup$
1

1 Answer 1

0
$\begingroup$

The currently dominant NMT paradigm is based on an encoder-decoder architecture where the translation is generated autoregressively. This means that each token in the translation is generated conditioning not only on the source sentence but also on the previously generated tokens. The encoder generates a representation of the source sentence in one go and then the decoder takes this representation and generates the target side tokens one by one. This way, the decoder is responsible for the autoregressive part.

When you have an architecture where there is no decoder, you have non-autoregressive (NAR) NMT. It is possible to build a NAR model, but most attempts to train models directly in such a way have failed. They usually repeat the same words once and again, utterly generating garbage instead of actual translations.

Nevertheless, if NAR NMT were possible, it would lead to huge speedups ($O(1)$ instead of $O(n)$ complexity). This is currently a very active line of research. Most of the proposed attempts rely on creating some sort of intermediate (latent) representation to feed to the decoder, which then decodes non-autoregressively.

These are some of the latest articles from that area:

$\endgroup$
1
  • $\begingroup$ Thank you very much. $\endgroup$
    – DunkOnly
    Sep 16, 2019 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.