I am trying to train an NMT model where the source side is roman text of Asian languages from social media, and target side is English. Note that since roman text is not native to Asia, the romanizations done by people to type on the Internet are very personal and hence a bit noisy, but easily intelligible to native speakers.
The following is an example for writing a Hindi sentence in different ways:
- Vaise bhi mere paas jo bhi hai maine aap ko sab kuch dey diyaa bhaai
- wesebi mr pas jobi h, mene apko sbkch dedia bhai
So I think sub-word tokenizers might not help much here (for the source side), and will also not be robust to different variations of noise. (Note that the target side could be sub-word tokenizer.)
What models and tokenizer for the source side is generally suggested and works for such cases? Would character-level models would be the best suited?