It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving language models it takes BERT as a based an adds some kind of fine-tuning or filtering or something. I don't understand why BERT became the default in research circles when all anyone hears about publicly is GPT-2,3,4 or more recently LLAMA-2. I have a feeling it has something to do with BERT being open-source, but that can't be the whole story. This question might not be specific enough, please let me know. Thanks.
There are many contributing factors to the abundance of research based on BERT vs the research based on Llama:
- Age: BERT has been around for far longer than Llama (2018 vs 2023), so it has more traction with researchers because it has been applied to many many things, so people know they work and has probably already been applied to a problem similar to yours.
- Computational resources: BERT is lightweight compared to Llama. Anyone can train BERT on a single medium-range GPU. To use Llama for inference you need a lot of very powerful GPUs, let alone training it. Most research groups have modest computational resources.
- Appropriateness for downstream tasks: BERT is easily applied to text classification because it has the output at the
[CLS]token position, which can be directly attached a classification head. Llama is an autoregressive language model, which makes it less obvious how to use it for classification. Of course, you can just approach the task at the natural language level and ask Llama to classify the input text, but the reliability of this kind of approach is not 100% and the model may just answer with their diatribe about why it's not Ok to proceed with your request. On the other hand, BERT is not meant as a generative model, you you'd better not use it for a generative task.
Although LLM's like GPT-3 and LLAMA have gain public attention due to marketing, BERT is the foundation of all Large Language Models being open-source and the first one to base on transformer architecture.
Also BERT's bidirectional context-aware embeddings allowed it to capture rich contextual information from both left and right contexts of a word. This property made BERT versatile and effective across a wide range of NLP tasks.
These are the reasons why in researchers use BERT instead of other LLMs