Training NMT models for noisy social media roman text

Question

I am trying to train an NMT model where the source side is roman text of Asian languages from social media, and target side is English. Note that since roman text is not native to Asia, the romanizations done by people to type on the Internet are very personal and hence a bit noisy, but easily intelligible to native speakers.

The following is an example for writing a Hindi sentence in different ways:

• Vaise bhi mere paas jo bhi hai maine aap ko sab kuch dey diyaa bhaai
• wesebi mr pas jobi h, mene apko sbkch dedia bhai

So I think sub-word tokenizers might not help much here (for the source side), and will also not be robust to different variations of noise. (Note that the target side could be sub-word tokenizer.)

What models and tokenizer for the source side is generally suggested and works for such cases? Would character-level models would be the best suited?

Answer 1

Character-level models are only rarely better than subwords, not even in situations where you would naturally expect it (cf. recent papers When is Char Better Than Subword: A Systematic Study of Segmentation Algorithms for Neural Machine Translation, Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems). The biggest gains can come from data handling, not likely from modeling improvements.

The SoTA in low-resource MT is using pre-trained models such as mBART or MASS, even in cases when the pre-training was done in different languages. In this case, you need to use the tokenization of the pre-trained model which is probably suboptimal, but the benefit of pre-training is usually bigger.

It would be very helpful if you can generate synthetic data by romanizing existing parallel corpora. Also, if you can monolingual data, iterative back-translation is going to help a lot.