Text generation is perhaps one of the fun things to do with old NGram or new BERT/ELMo models. I am wondering can BERT be used to generate text from the end of a sentence, or better in both directions. That is instead of giving some starting words, we give some ending words.
1
-
$\begingroup$ Similar to this question, but not the same IMO. $\endgroup$– noeDec 12, 2019 at 15:30
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Neither BERT nor ELMo can be used as-is for next word (or previous word) predictions. BERT is trained on a masked language model (LM) task and can therefore only be used to guess masked words, obtaining this way a contextual representation of them. There have been some attempts to use BERT for text generation, but they have been unsuccessful up to now. ELMo is a bidirectional LM, so you need the whole sentence to use it as input.
Both BERT an ELMo use the whole sentence (except masked tokens for BERT) by construction, so they can hardly be trained on "causal" LM tasks without modifying the models:
BERT would need to be added a self-attention mask to force causality. This would be equivalent to using BERT's pre-trained weights as initial values for a normal Transformer (causal) language model or any text-generation model, like a neural machine translation (NMT) one; actually, this has been attempted for NMT here and here.
ELMO should be stripped from one of the directions' LSTM. This would be equivalent to using one of the direction's parameters to initialize an LSTM for text generation.
Another option would be to use them for knowledge distillation. This has been done with BERT.
-
$\begingroup$ "BERT has a Mouth and must Speak, but it is not an MRF" does not, to the best of my understaning, claims that BERT cannot be used to successfully generate text; it just says that it could not be interpreted as a Markov Random Field. $\endgroup$ Jan 8, 2020 at 22:34
-
$\begingroup$ My understanding is that the MRF claim is what vertebrated the paper and, after proven not to be correct, the paper as a whole is to be put on hold, at least until it is reformulated under this new framing. Apart from that, the generations from Table 1 (left) shed some doubt on the model's ability to generate coherent text. $\endgroup$– noeJan 8, 2020 at 23:50