14
$\begingroup$

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?

$\endgroup$
  • $\begingroup$ Have you seen the original publication? It seems to be addressing prediction at the sentence level, as explained in its section 3.3.2. $\endgroup$ – mapto Feb 28 '19 at 9:07
  • $\begingroup$ Consider a related discussion on GitHub. $\endgroup$ – mapto Mar 13 '19 at 10:04
14
$\begingroup$

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling.

BERT is trained on a masked language modeling task and therefore you cannot "predict the next word". You can only mask a word and ask BERT to predict it given the rest of the sentence (both to the left and to the right of the masked word).

This way, with BERT you can't sample text like if it were a normal autoregressive language model. However, BERT can be seen as a Markov Random Field Language Model and be used for text generation as such. See article BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model for details. The authors released source code and a Google Colab notebook.

Update: the authors of the MRF article discovered their analysis was flawed and BERT is not a MRF, see this

| improve this answer | |
$\endgroup$
  • $\begingroup$ Results do not look very mature :( $\endgroup$ – Itachi Mar 1 '19 at 9:29
  • 2
    $\begingroup$ Why can't you just control the mask to be the last word in the sequence? Then use BERT to predict based the masked token (next word). I'm still digesting these results so I can't guide how to implement. Still, seems like a plausible approach. $\endgroup$ – Sledge Mar 5 '19 at 21:40
  • $\begingroup$ This was tried by some guy on the several twitter discussions about BERT after it was released and he confirmed that BERT failed with the approach @Sledge is describing.If you cut a sentence and ask BERT to predict the next word, it will not be able to use the right-hand part of the sentence, which it needs to perform the prediction. $\endgroup$ – ncasas Mar 6 '19 at 13:55
  • $\begingroup$ I see, @ncasas thanks for the explanation. $\endgroup$ – Sledge Mar 6 '19 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.