Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Is it possible to feed embeddings from XLM- RoBERTa to transformer seq2seq model? I'm working on NMT that translates verbal language sentences to sign language sentences (e.g Input: He sells food. Output (sign language sentence): Food he sells). But I have a very small dataset of sentence pairs - around 1000. And the language is a low resource language.
I am a new researcher on the field of deep learning. Please help with your valuable advice.
It is indeed possible, but the question is if it is a good idea.
FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and replace the encoder with XLM-R.
Another option would be using Huggingface's Transofrmers that also provides basic support for sequence-to-sequence models as they show in their blog post.
Now, why I don't think it is a good idea. Recent papers show that:
Improving MT with pre-trained Transformers requires a lot of effort and it is still questionable whether it pays off, see Incorporating BERT into Neural Machine Translation .
The encoder of MT capture different structures: see [What does BERT look at?](What does BERT look at?) and Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned.
In an end-to-end trained translation system, a large part of the translation process is happening already in the encoder, see Analyzing Word Translation of Transformer Layers.
I assume you only have little training data and therefore you hope that pre-trained representation might help, but I believe you should put more effort in optimizing an end-to-end model to work in low-resource setup (have a look at Revisiting Low-Resource Neural Machine Translation: A Case Study) and data augmentation technique like iterative back-translation that it used in unsupervised machine translation.
