I want to work on an unsupervised machine translation system using a variational autoencoder. I did a literature review but didn't find any related work, and most of the work is based on denoising autoencoders. There are a few questions regarding this problem -
- What kind of encoding to use for this machine translation system? (like BPE, wordpiece, etc)
- What kind of encoder-decoder model would be employed?
- What will be the latent variable's expected loss value, and how to define it? Hint: Although JS-divergence and Energy distance are both smooth functions and differentiable, KL-divergence is not.
- Most text-VAE-based models do not support standard training methods such as backpropagation, what are some other approaches?
I thought maybe we could use variational autoencoders for learning distributions from two monolingual corpora for unsupervised translation. Each autoencoder will learn a distribution for the respective language, however, I also need to build a common latent space between the two languages. I'm not very sure how to align the spaces of the two languages.
Looking for some responses. It'd be nice to have some good mathematical formulations as well.