I was trying to get a grasp about BERT and found this post in DS StackExchange:

Can BERT do the next-word-predict task?

In broad terms, it says that Bert cannot be used for next-word prediction. I suppose that next-word prediction it could be used, for example, in some sort of autocorrect tools. However, I saw in this blog:


That BERT could be used for text prediction; so, why it is said that it cannot be used for next-word prediction? If somebody could explain the caveats of it and an example to clarify this, it would be really helpful.



1 Answer 1


You can also take a random word generator and use it to "generate text". Does it generate text? Well, yes, but no. The same applies to BERT.

The problem with using BERT for text generation is that it is not meant for that. BERT is a Transformer encoder. It is meant to receive as input a whole piece of text, and it will generate one output per input token (trying to guess any tokens in the input masked with the [MASK] token) plus a sentence-level output (the output at the first position, which receives the [CLS] token). Therefore, BERT is not trained to receive only the previous tokens and generate the next one.

So to answer your question: of course, you can use BERT to generate text by placing a [MASK] token at the end of a sequence of tokens and using its prediction to autoregressively build more text, but don't expect a good output, at least not on the level of a "causal" language model like GPT-2 or GPT-3, which are Transformer decoders trained specifically to generate the next token based on the previous ones.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.