As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?

  • 1
    $\begingroup$ Have you seen the original publication? It seems to be addressing prediction at the sentence level, as explained in its section 3.3.2. $\endgroup$
    – mapto
    Commented Feb 28, 2019 at 9:07
  • $\begingroup$ Consider a related discussion on GitHub. $\endgroup$
    – mapto
    Commented Mar 13, 2019 at 10:04

1 Answer 1


BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling.

BERT is trained on a masked language modeling task and therefore you cannot "predict the next word". You can only mask a word and ask BERT to predict it given the rest of the sentence (both to the left and to the right of the masked word).

This way, with BERT you can't sample text like if it were a normal autoregressive language model. However, BERT can be seen as a Markov Random Field Language Model and be used for text generation as such. See article BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model for details. The authors released source code and a Google Colab notebook.

Update: the authors of the MRF article discovered their analysis was flawed and BERT is not a MRF, see this

Update 2: despite not being meant for next word prediction, there have been attempts at using BERT that way. Here you can find a project that does next word prediction with BERT, XLNet, RoBERTa, etc.

  • 3
    $\begingroup$ Why can't you just control the mask to be the last word in the sequence? Then use BERT to predict based the masked token (next word). I'm still digesting these results so I can't guide how to implement. Still, seems like a plausible approach. $\endgroup$
    – Sledge
    Commented Mar 5, 2019 at 21:40
  • 2
    $\begingroup$ This was tried by some guy on the several twitter discussions about BERT after it was released and he confirmed that BERT failed with the approach @Sledge is describing.If you cut a sentence and ask BERT to predict the next word, it will not be able to use the right-hand part of the sentence, which it needs to perform the prediction. $\endgroup$
    – noe
    Commented Mar 6, 2019 at 13:55
  • $\begingroup$ I see, @ncasas thanks for the explanation. $\endgroup$
    – Sledge
    Commented Mar 6, 2019 at 15:12
  • $\begingroup$ @ncasas May I ask if this statement is (BERT not suitable for next word prediction) is also true for transformer architecture at general? Because I got impression that gpt utilize next token prediction, but I'm not completely sure. $\endgroup$
    – viceriel
    Commented Dec 21, 2020 at 14:09
  • $\begingroup$ No, the Transformer architecture was originally meant for text generation. GPT-2 is a traditional Transformer decoder trained on a causal language model loss, and therefore it is suitable for text generation. On the other hand, BERT is trained on a masked language model loss, and that is why it is not meant for text generation. $\endgroup$
    – noe
    Commented Dec 21, 2020 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.