Questions tagged [bert]
BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers
357
questions
67
votes
4
answers
78k
views
What is purpose of the [CLS] token and why is its encoding output important?
I am reading this article on how to use BERT by Jay Alammar and I understand things up until:
For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we ...
- 887
43
votes
2
answers
31k
views
What is GELU activation?
I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as
$$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which in turn is approximated to $$0.5x(1 + tanh[\sqrt{
2/π}(x + ...
- 2,395
38
votes
7
answers
73k
views
27
votes
5
answers
19k
views
BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?
Word2vec:
Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are ...
- 521
24
votes
7
answers
22k
views
Why is the decoder not a part of BERT architecture?
I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT ...
21
votes
1
answer
11k
views
Can BERT do the next-word-predict task?
As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
- 691
16
votes
2
answers
21k
views
What is the use of [SEP] in paper BERT?
I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun.
However, I have a question.
If I have 2 sentences, which are s1 and s2, and our fine-tuning ...
- 161
14
votes
2
answers
8k
views
Preprocessing for Text Classification in Transformer Models (BERT variants)
This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models?
I remember for training a Word2Vec or Glove,...
- 4,207
13
votes
2
answers
13k
views
What are the good parameter ranges for BERT hyperparameters while finetuning it on a very small dataset?
I need to finetune BERT model (from the huggingface repository) on a sentence classification task. However, my dataset is really small.I have 12K sentences and only 10% of them are from positive ...
- 249
12
votes
1
answer
7k
views
What is whole word masking in the recent BERT model?
I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it:
In the original pre-...
- 223
12
votes
1
answer
13k
views
what is the first input to the decoder in a transformer model?
The image is from url: Jay Alammar on transformers
K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each ...
- 574
11
votes
2
answers
12k
views
Does BERT has any advantage over GPT3?
I have read a couple of documents that explain in detail about the greater edge that GPT-3(Generative Pre-trained Transformer-3) has over BERT(Bidirectional Encoder Representation from Transformers). ...
- 213
10
votes
2
answers
3k
views
Why should I understand AI architectures?
Why should I understand what is happening deep down in some AI architecture?
For example LSTM-BERT- Partial Conv... Architectures like this. Why should I understand what is going on while I can find ...
- 117
9
votes
2
answers
3k
views
Is BERT a language model?
Is BERT a language model in the sense of a function that gets a sentence and returns a probability?
I know its main usage is sentence embedding, but can it also provide this functionality?
- 786
9
votes
1
answer
5k
views
Bert Fine Tuning with additional features
I want to use Bert for an nlp task. But I also have additional features that I would like to include.
From what I have seen, with fine tuning, one only changes the labels and retrains the ...
- 193
8
votes
2
answers
7k
views
How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)
First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library....
- 805
8
votes
2
answers
16k
views
What is the difference between BERT and Roberta
I want to understand the difference between BERT and Roberta. I saw the article below.
https://towardsdatascience.com/bert-roberta-distilbert-xlnet-which-one-to-use-3d5ab82ba5f8
It mentions that ...
- 183
8
votes
1
answer
748
views
Why is word prediction an obsession in Natural Language Processing?
I have heard how great BERT is at masked word prediction, i.e. predicting a missing word from a sentence.
In a Medium post about BERT, it says:
The basic task of a language model is to predict ...
- 183
8
votes
3
answers
5k
views
Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?
I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
- 415
8
votes
1
answer
8k
views
BERT vs GPT architectural, conceptual and implemetational differences
In the BERT paper, I learnt that BERT is encoder-only model, that is it involves only transformer encoder blocks.
In the GPT paper, I learnt that GPT is decoder-only model, that is it involves only ...
- 205
8
votes
2
answers
3k
views
What should be the labels for subword tokens in BERT for NER task?
For any NER task, we need a sequence of words and their corresponding labels.
To extract features for these words from BERT, they need to be tokenized into subwords.
For example, the word ...
- 374
7
votes
2
answers
5k
views
Does BERT use GLoVE?
From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated.
What I don't know ...
- 173
7
votes
1
answer
1k
views
Do transformers (e.g. BERT) have an unlimited input size?
There are various sources on the internet that claim that BERT has a fixed input size of 512 tokens (e.g. this, this, this, this ...). This magical number also appears in the BERT paper (Devlin et al. ...
- 193
7
votes
4
answers
7k
views
BERT: it is possible to use it for topic modeling?
I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
- 193
7
votes
3
answers
3k
views
meaning of fine-tuning in nlp task
There are two types of transfer learning model. One is
feature extraction, where the weights of the pre-trained model are not changed while training on the actual task and other is the weights of the ...
- 521
7
votes
1
answer
9k
views
How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?
I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
- 696
6
votes
1
answer
7k
views
How can I add custom numerical features for training to BERT fine tuning?
I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy.
My original dataset consists of a pair of sentences (like MRPC ...
- 61
6
votes
1
answer
9k
views
What GPU size do I need to fine tune BERT base cased?
I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
6
votes
1
answer
741
views
Why do BERT classification do worse with longer sequence length?
I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts ...
- 207
5
votes
1
answer
11k
views
How pre-trained BERT model generates word embeddings for out of vocabulary words?
Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
- 2,041
5
votes
2
answers
2k
views
how to run bert's pretrained model word embeddings faster?
I'm trying to get word embeddings for clinical data using microsoft/pubmedbert.
I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, ...
- 138
5
votes
2
answers
27k
views
How to load the pre-trained BERT model from local/colab directory?
Hi i downloaded the BERT pretrained model (https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip) from here and saved to a directory in gogole colab and in local .
when i ...
- 1,431
5
votes
1
answer
7k
views
Calculating cosine similarity between 3D arrays using Python
I have two matrices with multiple columns and three rows each. I calculated the cosine similarity (sklearn) but it gives the result as a matrix. How can I obtain one single value?
The matrices are the ...
5
votes
3
answers
10k
views
Generating synonyms or similar words from multiples word embeddings
I am looking for a way to generate synonyms, using word embeddings. From one word, and from multiple words. Such as the two example below:
"word" -> Word embedding -> generate synonym of "word"
"...
- 151
5
votes
1
answer
2k
views
Is a BiLSTM layer required if we use BERT?
I am new to Deep learning based NLP and I have a doubt -
I am trying to build a NER model and I found some journals where people are relying on BERT-BiLSTM-CRF model for it. As far as I know BERT is a ...
5
votes
2
answers
9k
views
How to add a CNN layer on top of BERT?
I am just playing with bert (Bidirectional Encoder Representation from Transformer)
Research Paper
Suppose I want to add any other model or layers like Convolutional Neural Network layers (CNN), Non ...
- 51
5
votes
3
answers
7k
views
where to store embeddings for similarity search?
I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
- 51
5
votes
1
answer
1k
views
BertPunc (punctuation restoration with BERT)
I've found the script for punctuation restoration. And I have one question about this method.
I will briefly explain the logic of the author. One of four tokens is assigned for each word: Other (0), ...
- 308
5
votes
2
answers
80
views
User profiling based on multiple posts
I currently have collected a dataset of different social media posts for each user with labels assigned to each user. I tried to use LSTM, and BERT for the text classification problem, So for each ...
- 51
4
votes
1
answer
978
views
Bert for QuestionAnswering input exceeds 512
I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long ...
- 242
4
votes
2
answers
3k
views
Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?
It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving ...
- 143
4
votes
1
answer
442
views
BERT word embedings for finding word definition
Can BERT, GPT or other contextualised embedings be used for finding word definitions? What would be the most effective and not complicated approach for tackling a sample task as described below.
Map ...
- 51
4
votes
2
answers
8k
views
Can we use BERT for only word embedding and then use SVM/RNN to do intent classification?
According to this article, "Systems used for intent classification contain the following two components:
Word embedding, and a classifier." This article also evaluated BERT+SVM and Word2Vec+...
- 41
4
votes
1
answer
1k
views
NLP Transformers: How to get a fixed sentences embedding vectors size?
I'm loading a language model from torch hub (CamemBERT a French RoBERTa-based model) and using it do embed some sentences:
...
- 141
3
votes
1
answer
12k
views
What is a 'hidden state' in BERT output?
I'm trying to understand the workings and output of BERT, and I'm wondering how/why each layer of BERT has a 'hidden state'.
I understand what RNN's have a 'hidden state' that gets passed to each ...
- 213
3
votes
2
answers
3k
views
What are the elements in a BERT word embedding?
As far as I understand, BERT is a word embedding that can be fine-tuned or used directly.
With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one ...
- 179
3
votes
2
answers
6k
views
Similarity of words using BERTMODEL
I want to find the similarity of words using the BERT model within the NER task. I have my own dataset so, I don't want to use the pre-trained model.
I do the following:
...
- 41
3
votes
2
answers
2k
views
What is the difference between BERT architecture and vanilla Transformer architecture
I'm doing some research for the summarization task and found out BERT is derived from the Transformer model. In every blog about BERT that I have read, they focus on explaining what is a bidirectional ...
- 133
3
votes
2
answers
281
views
NER - What advantage does IO Format have over BIO Format
In this paper, the authors say that they used IO schema instead of BIO in their dataset, which, if I am not wrong, means they just tag the corresponding Entity Type or "O" in case the word ...
- 33
3
votes
1
answer
2k
views
BERT - The purpose of summing token embedding, positional embedding and segment embedding
I read the implementation of BERT inputs processing (image below). My question is why the author chose to sum up three types of embedding (token embedding, positional embedding and segment embedding)?
- 133