2
$\begingroup$

I'm working on neural machine translator that translates English sentences to American sign language sentences(e.g below). I've a quite small dataset - around 1000 sentence pairs. I'm wondering if it is possible to fine-tune BERT, ELMO or XLnet for Seq2seq encoder/decoder machine translation.

English: He sells food.

American sign language: Food he sells

Improve this question
$\endgroup$
4
  • $\begingroup$ I don't know American sign language, but is it only about word reordering ? $\endgroup$
    – Astariul
    Feb 25, 2020 at 3:33
  • $\begingroup$ Not only reordering. the grammar is a bit different than English. English relies on Subject-Verb-Object (SVO) sentence structure, while ASL more frequently uses Topic-Comment structure. $\endgroup$
    – NLP Dude
    Feb 25, 2020 at 6:48
  • 1
    $\begingroup$ Then it seems to be very similar to Translation task. Bert is only an encoder, so it cannot be used alone for Seq2Seq tasks, but it's definitely possible to add a decoder and use Bert as encoder. Or simply used seq2seq architecture such as BART. $\endgroup$
    – Astariul
    Feb 25, 2020 at 7:18
  • $\begingroup$ Thank you so much, It's very helpful. one last question, Is it possible to do something similar with guru99.com/seq2seq-model.html and use BERT as an encoder. $\endgroup$
    – NLP Dude
    Feb 25, 2020 at 11:20

1 Answer 1

Reset to default
1
$\begingroup$

You can view models like ELMo or BERT to be encoder-only. They can be easily used for classification or sequence tagging, but the tag sequence is typically monotonically aligned with the source sequence. Even though the Transformer layers in BERT or XLNet are in theory capable of arbitrary reordering (which is used in non-autoregressive machine translation models), this is not what BERT or XLNet were trained for and therefore it will be hard to finetune for that.

If at least the vocabulary is the same on both the source and target side, I would recommend pre-trained sequence-to-sequence models: MASS or BART.

If the both the grammar and vocabulary and grammar of the sign language are quite different, maybe using BERT as an encoder and training your own lightweight autoregressive decoder might be the correct way.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you so much, It's very helpful. one last question, Is it possible to do something similar with guru99.com/seq2seq-model.html and use BERT as an encoder. I'm thinking pretrain BERT from scratch and to do this for low resource language rather than English to ASL. $\endgroup$
    – NLP Dude
    Feb 25, 2020 at 11:20
  • $\begingroup$ Yes, it should be possible. However, I would consider using a Transformer decoder instead of GRU, but both should work. $\endgroup$
    – Jindřich
    Feb 25, 2020 at 13:27
  • $\begingroup$ Thank you so much, I really appreciate it. $\endgroup$
    – NLP Dude
    Feb 25, 2020 at 13:41
  • $\begingroup$ I was researching on cross-lingual approach and I've found that Facebook AI research team released XLM-R which is trained in one language and used with other languages. I would like to ask you If It's possible to finetune XLM-R for the task we discussed. $\endgroup$
    – NLP Dude
    Feb 28, 2020 at 9:14
  • $\begingroup$ Couldn't you use BERT's next sentence prediction task as a basis on which to fine-tune your seq2seq downstream task? $\endgroup$
    – npit
    Sep 21, 2020 at 2:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.