why some authors said that BERT cannot be used for text prediction?

Question

I was trying to get a grasp about BERT and found this post in DS StackExchange:

Can BERT do the next-word-predict task?

In broad terms, it says that Bert cannot be used for next-word prediction. I suppose that next-word prediction it could be used, for example, in some sort of autocorrect tools. However, I saw in this blog:

https://medium.com/mlearning-ai/an-illustration-of-next-word-prediction-with-state-of-the-art-network-architectures-like-bert-gpt-c0af02921f17

That BERT could be used for text prediction; so, why it is said that it cannot be used for next-word prediction? If somebody could explain the caveats of it and an example to clarify this, it would be really helpful.

Thanks.

Answer 1

You can also take a random word generator and use it to "generate text". Does it generate text? Well, yes, but no. The same applies to BERT.

The problem with using BERT for text generation is that it is not meant for that. BERT is a Transformer encoder. It is meant to receive as input a whole piece of text, and it will generate one output per input token (trying to guess any tokens in the input masked with the [MASK] token) plus a sentence-level output (the output at the first position, which receives the [CLS] token). Therefore, BERT is not trained to receive only the previous tokens and generate the next one.

So to answer your question: of course, you can use BERT to generate text by placing a [MASK] token at the end of a sequence of tokens and using its prediction to autoregressively build more text, but don't expect a good output, at least not on the level of a "causal" language model like GPT-2 or GPT-3, which are Transformer decoders trained specifically to generate the next token based on the previous ones.