It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving language models it takes BERT as a based an adds some kind of fine-tuning or filtering or something. I don't understand why BERT became the default in research circles when all anyone hears about publicly is GPT-2,3,4 or more recently LLAMA-2. I have a feeling it has something to do with BERT being open-source, but that can't be the whole story. This question might not be specific enough, please let me know. Thanks.

  • $\begingroup$ Can you elaborate on particular research paper(s) that you think would better suited to be using an alternative model? These models all have advantages/disadvantages that depend on the task, so I'm not sure if I'd say that BERT is considered the default model for research. $\endgroup$ Aug 3 at 13:44

2 Answers 2


There are many contributing factors to the abundance of research based on BERT vs the research based on Llama:

  • Age: BERT has been around for far longer than Llama (2018 vs 2023), so it has more traction with researchers because it has been applied to many many things, so people know they work and has probably already been applied to a problem similar to yours.
  • Computational resources: BERT is lightweight compared to Llama. Anyone can train BERT on a single medium-range GPU. To use Llama for inference you need a lot of very powerful GPUs, let alone training it. Most research groups have modest computational resources.
  • Appropriateness for downstream tasks: BERT is easily applied to text classification because it has the output at the [CLS] token position, which can be directly attached a classification head. Llama is an autoregressive language model, which makes it less obvious how to use it for classification. Of course, you can just approach the task at the natural language level and ask Llama to classify the input text, but the reliability of this kind of approach is not 100% and the model may just answer with their diatribe about why it's not Ok to proceed with your request. On the other hand, BERT is not meant as a generative model, you you'd better not use it for a generative task.
  • $\begingroup$ How often are generative tasks necessary for research, outside of creating a new generative model? $\endgroup$
    – Ethan
    Aug 3 at 18:43
  • $\begingroup$ There are lines of research that are generative in nature, e.g. machine translation, some types of summarization, while some others are not, e.g. text classification, POS tagging. There are plenty of both types. $\endgroup$
    – noe
    Aug 3 at 19:16

Although LLM's like GPT-3 and LLAMA have gain public attention due to marketing, BERT is the foundation of all Large Language Models being open-source and the first one to base on transformer architecture.

Also BERT's bidirectional context-aware embeddings allowed it to capture rich contextual information from both left and right contexts of a word. This property made BERT versatile and effective across a wide range of NLP tasks.

These are the reasons why in researchers use BERT instead of other LLMs


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.