Unsupervised Machine Translation System Using Variational Autoencoder Models

Question

I want to work on an unsupervised machine translation system using a variational autoencoder. I did a literature review but didn't find any related work, and most of the work is based on denoising autoencoders. There are a few questions regarding this problem -

1. What kind of encoding to use for this machine translation system? (like BPE, wordpiece, etc)
2. What kind of encoder-decoder model would be employed?
3. What will be the latent variable's expected loss value, and how to define it? Hint: Although JS-divergence and Energy distance are both smooth functions and differentiable, KL-divergence is not.
4. Most text-VAE-based models do not support standard training methods such as backpropagation, what are some other approaches?

I thought maybe we could use variational autoencoders for learning distributions from two monolingual corpora for unsupervised translation. Each autoencoder will learn a distribution for the respective language, however, I also need to build a common latent space between the two languages. I'm not very sure how to align the spaces of the two languages.

Looking for some responses. It'd be nice to have some good mathematical formulations as well.

Answer 1

First I will answer your questions:

1. The tokenization strategy is constrained by how the alignment is done. If you need word-level representations for the alignment, then the tokenization should be at word-level.

2. For the encoder part: either an LSTM or a Transformer encoder (like BERT), where all the input sentence is condensed on a fixed-size representation. For the decoder part: either an LSTM or a Transformer decoder.

3. This is a design decision. You choose. However, based on the literature (see this), I would suggest normalizing flows.

4. You state that "Most text-VAE-based models do not support standard training methods such as backpropagation". I don't agree with that. While this is true for Generative Adversarial Networks (GAN) due to the discrete junction between generator and discriminator, in the case of textual VAEs there is no such problem. The problem with textual VAEs is the need to cram all the information from the input sentence in a fixed-sized representation. This is the same problem that the first LSTM-based attention-less encoder-decoder machine translation models had. Bahdanau's and Luong's attention came to mitigate this problem. If you want to obtain competitive results, you would also need a mechanism that can give the decoder access to more than a fixed-length representation, either attention or something else.

Aligning word representation spaces is the base of most unsupervised machine translation approaches. It is challenging, and the results are only good enough to bootstrap a back-translation loop. My guess is that aligning sentence representation spaces is much more difficult, because you cannot apply the same assumptions as with the word-level spaces

Probably the most promising things would be to build on top of the ideas of the variational machine translation approaches proposed on the literature (e.g. this). However, given that a supervised approach to variational machine translation did not take off, I have doubts that an unsupervised one would give good results.