I'm working on NMT model which the input and the target sentences are from the same language (but the grammar differs). I'm planning to pre-train and use BERT since I'm working on small dataset and low/under resource language. so is it possible to feed BERT to the seq2Seq encoder/decoder?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
Sure, why not? An encoder/decoder is basically agnostic to the format of the token vectors, whether they be derived via Word2Vec, BERT, GPT2, etc.
The more challenging aspect of this, should you not have figured it out already, might be finding a pretrained embedding model for your low resource language. Given a small dataset, training your own from scratch seems infeasible. You could potentially find a BERT pretrained over a similar language (e.g. within the same family or grammatical structure) and fine-tune the last layer according to your dataset. Conventional BERT is trained over the English language, with some BERTs spinning off in other languages.
-
$\begingroup$ Thank you Alex.In order to train BERT model, I've scraped around 2 million sentences from the web and cleaned the data. But for the second task(input and the target sentences for seq2seq ), I've got a quite small dataset - around 1000 sentence pairs. similar to guru99.com/seq2seq-model.html but using a language model. Have you seen any examples/research-work that you can share with me? $\endgroup$– NLP DudeCommented Feb 22, 2020 at 22:11
-
$\begingroup$ I don't think I understand what your issue is. Are you confused about how to embed tokens/sentences or how to use embeddings in a seq2seq model? $\endgroup$– Alex LCommented Feb 23, 2020 at 0:36
-
$\begingroup$ I'm confused about how to use BERT Word embeddings in a seq2seq model. $\endgroup$– NLP DudeCommented Feb 23, 2020 at 6:28