8
votes
Accepted
Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?
There are many contributing factors to the abundance of research based on BERT vs the research based on Llama:
Age: BERT has been around for far longer than Llama (2018 vs 2023), so it has more ...
4
votes
Why is the decoder not a part of BERT architecture?
In short, Bidirectional Encoder Representations from Transformers (BERT) is not designed for decorder-related tasks.
I can't see how BERT makes predictions without using a decoder unit, which was a ...
3
votes
Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?
Adding/complementing the other answers, BERT gives the possibility to access/obtain the embeddings of the fed input (which wasn't and still isn't the case of some other models).
The embeddings are ...
2
votes
Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?
Although LLM's like GPT-3 and LLAMA have gain public attention due to marketing, BERT is the foundation of all Large Language Models being open-source and the first one to base on transformer ...
2
votes
Accepted
Why the standard deviation of the BERT weight initialization is 0.02 by default
According to the article Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention [Zhang et al., 2019], the Transformer architecture suffers from poor convergence due to ...
1
vote
Reducing emails token count preprocessing for Large Email Datasets - Feeding LLMs
Instead of the approaches you mentioned, I suggest a completely different approach: a retrieval-augmented generation (RAG) system. I am doing this because what you described is a typical use case for ...
1
vote
Accepted
How can I use contextual embeddings with BERT for sentiment analysis/classification
For this kind of setup, you should use the output at the first position and train a linear classifier over your 3 labels.
BERT was trained with inputs that were prepended a special token ...
1
vote
Accepted
Is it methodologically correct to use the data to be used for finetuning in the pretrain phase of the BERT model?
I'd say that it's correct.
BERT pre-training doesn't use labels, as it uses two self-supervised objectives:
masked language model (mask a word in the middle of a sentence, and guess what it is)
next ...
1
vote
Accepted
How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder
First, note that the purpose of next sentence prediction objective is not to contribute to the contextual embeddings part, but to allow other downstream tasks like sentence classification and textual ...
1
vote
Accepted
Training model using BERT
You are overfitting A LOT.
This is usual when finetuning BERT on small datasets. I suggest you take a look at the BERT article to use it as a guidance for sensible hyperparameter values and finetuning ...
1
vote
building embeddings for Phrases from scratch
If you want to train on phrases, then you will have to devise your tokenizer that way.
Pseudo code will look like:
...
1
vote
building embeddings for Phrases from scratch
There are several ways word-embeddings are trained, however most of them require a ton of data. They usually involve learning vector representations that are useful for some self-supervised objective, ...
1
vote
Accepted
Building BERT tokenizer with custom data
Yes, of course, it is possible, but that implies that you need to train again the whole model, not only the tokenizer.
Here you can find the steps to do everything with the HuggingFace libraries, ...
1
vote
what is the difference between NSP and text prediction
NSP is not for predicting the next sentence but a binary classification task to predict whether the second sentence was originally immediately after the first or not:
During BERT's training, it is ...
1
vote
detect/generate possible tokens for the dataset (name,type/category,signatures)
Given the nature of the problem, it might not be amenable to machine learning. The structure of the data drives how it can be modeled. The features are "name" (assumed to be a string) and &...
Only top scored, non community-wiki answers of a minimum length are eligible
Related Tags
bert × 355nlp × 211
transformer × 93
machine-learning × 51
deep-learning × 51
word-embeddings × 45
python × 28
text-classification × 27
huggingface × 26
pytorch × 23
language-model × 20
tensorflow × 19
finetuning × 15
neural-network × 14
sentiment-analysis × 13
lstm × 12
word2vec × 11
transfer-learning × 11
embeddings × 11
attention-mechanism × 11
classification × 10
keras × 10
named-entity-recognition × 9
gpt × 9
tokenization × 8