Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
71 votes
4 answers
93k views

What is purpose of the [CLS] token and why is its encoding output important?

I am reading this article on how to use BERT by Jay Alammar and I understand things up until: For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we ...
user3768495's user avatar
43 votes
2 answers
32k views

What is GELU activation?

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as $$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which in turn is approximated to $$0.5x(1 + tanh[\sqrt{ 2/π}(x + ...
thanatoz's user avatar
  • 2,425
39 votes
7 answers
77k views

How to get sentence embedding using BERT?

How to get sentence embedding using BERT? ...
star's user avatar
  • 1,481
29 votes
7 answers
25k views

Why is the decoder not a part of BERT architecture?

I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT ...
Hrishikesh Athalye's user avatar
27 votes
5 answers
20k views

BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?

Word2vec: Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are ...
sovon's user avatar
  • 521
21 votes
1 answer
12k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
CoderOnly's user avatar
  • 711
18 votes
2 answers
24k views

What is the use of [SEP] in paper BERT?

I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun. However, I have a question. If I have 2 sentences, which are s1 and s2, and our fine-tuning ...
xiangqing shen's user avatar
15 votes
2 answers
17k views

What are the good parameter ranges for BERT hyperparameters while finetuning it on a very small dataset?

I need to finetune BERT model (from the huggingface repository) on a sentence classification task. However, my dataset is really small.I have 12K sentences and only 10% of them are from positive ...
zwlayer's user avatar
  • 269
14 votes
2 answers
8k views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a Word2Vec or Glove,...
TwinPenguins's user avatar
  • 4,279
12 votes
1 answer
8k views

What is whole word masking in the recent BERT model?

I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it: In the original pre-...
kee's user avatar
  • 223
12 votes
1 answer
15k views

what is the first input to the decoder in a transformer model?

The image is from url: Jay Alammar on transformers K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each ...
mLstudent33's user avatar
11 votes
2 answers
12k views

Does BERT has any advantage over GPT3?

I have read a couple of documents that explain in detail about the greater edge that GPT-3(Generative Pre-trained Transformer-3) has over BERT(Bidirectional Encoder Representation from Transformers). ...
Bipin's user avatar
  • 213
10 votes
2 answers
3k views

Why should I understand AI architectures?

Why should I understand what is happening deep down in some AI architecture? For example LSTM-BERT- Partial Conv... Architectures like this. Why should I understand what is going on while I can find ...
CanP's user avatar
  • 117
10 votes
2 answers
20k views

What is the difference between BERT and Roberta

I want to understand the difference between BERT and Roberta. I saw the article below. https://towardsdatascience.com/bert-roberta-distilbert-xlnet-which-one-to-use-3d5ab82ba5f8 It mentions that ...
Noman Tanveer's user avatar
10 votes
1 answer
9k views

BERT vs GPT architectural, conceptual and implemetational differences

In the BERT paper, I learnt that BERT is encoder-only model, that is it involves only transformer encoder blocks. In the GPT paper, I learnt that GPT is decoder-only model, that is it involves only ...
Rnj's user avatar
  • 225
9 votes
2 answers
4k views

Is BERT a language model?

Is BERT a language model in the sense of a function that gets a sentence and returns a probability? I know its main usage is sentence embedding, but can it also provide this functionality?
Amit Keinan's user avatar
9 votes
1 answer
5k views

Bert Fine Tuning with additional features

I want to use Bert for an nlp task. But I also have additional features that I would like to include. From what I have seen, with fine tuning, one only changes the labels and retrains the ...
Jeff's user avatar
  • 193
8 votes
2 answers
7k views

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library....
fractalnature's user avatar
8 votes
3 answers
6k views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
Aaditya ura's user avatar
8 votes
1 answer
793 views

Why is word prediction an obsession in Natural Language Processing?

I have heard how great BERT is at masked word prediction, i.e. predicting a missing word from a sentence. In a Medium post about BERT, it says: The basic task of a language model is to predict ...
SamR's user avatar
  • 183
8 votes
2 answers
3k views

What should be the labels for subword tokens in BERT for NER task?

For any NER task, we need a sequence of words and their corresponding labels. To extract features for these words from BERT, they need to be tokenized into subwords. For example, the word ...
PinkBanter's user avatar
7 votes
3 answers
9k views

Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?

It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving ...
Ethan's user avatar
  • 173
7 votes
2 answers
5k views

Does BERT use GLoVE?

From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated. What I don't know ...
birdmw's user avatar
  • 173
7 votes
1 answer
2k views

Do transformers (e.g. BERT) have an unlimited input size?

There are various sources on the internet that claim that BERT has a fixed input size of 512 tokens (e.g. this, this, this, this ...). This magical number also appears in the BERT paper (Devlin et al. ...
Mew's user avatar
  • 233
7 votes
1 answer
7k views

How can I add custom numerical features for training to BERT fine tuning?

I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy. My original dataset consists of a pair of sentences (like MRPC ...
Ishita Gupta's user avatar
7 votes
4 answers
7k views

BERT: it is possible to use it for topic modeling?

I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
xcsob's user avatar
  • 193
7 votes
3 answers
3k views

meaning of fine-tuning in nlp task

There are two types of transfer learning model. One is feature extraction, where the weights of the pre-trained model are not changed while training on the actual task and other is the weights of the ...
sovon's user avatar
  • 521
7 votes
1 answer
9k views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
Harman's user avatar
  • 706
6 votes
1 answer
10k views

What GPU size do I need to fine tune BERT base cased?

I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
Darshan Bhandari's user avatar
6 votes
1 answer
900 views

Why do BERT classification do worse with longer sequence length?

I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts ...
Hooked's user avatar
  • 207
5 votes
1 answer
12k views

How pre-trained BERT model generates word embeddings for out of vocabulary words?

Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
Sayali Sonawane's user avatar
5 votes
2 answers
2k views

how to run bert's pretrained model word embeddings faster?

I'm trying to get word embeddings for clinical data using microsoft/pubmedbert. I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, ...
Madhur Yadav's user avatar
5 votes
2 answers
28k views

How to load the pre-trained BERT model from local/colab directory?

Hi i downloaded the BERT pretrained model (https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip) from here and saved to a directory in gogole colab and in local . when i ...
star's user avatar
  • 1,481
5 votes
1 answer
7k views

Calculating cosine similarity between 3D arrays using Python

I have two matrices with multiple columns and three rows each. I calculated the cosine similarity (sklearn) but it gives the result as a matrix. How can I obtain one single value? The matrices are the ...
GAYATRI VENUGOPAL's user avatar
5 votes
3 answers
11k views

Generating synonyms or similar words from multiples word embeddings

I am looking for a way to generate synonyms, using word embeddings. From one word, and from multiple words. Such as the two example below: "word" -> Word embedding -> generate synonym of "word" "...
David N's user avatar
  • 151
5 votes
1 answer
2k views

Is a BiLSTM layer required if we use BERT?

I am new to Deep learning based NLP and I have a doubt - I am trying to build a NER model and I found some journals where people are relying on BERT-BiLSTM-CRF model for it. As far as I know BERT is a ...
Saikat Bhattacharya's user avatar
5 votes
2 answers
9k views

How to add a CNN layer on top of BERT?

I am just playing with bert (Bidirectional Encoder Representation from Transformer) Research Paper Suppose I want to add any other model or layers like Convolutional Neural Network layers (CNN), Non ...
dev's user avatar
  • 51
5 votes
3 answers
7k views

where to store embeddings for similarity search?

I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
Angus's user avatar
  • 51
5 votes
1 answer
1k views

BertPunc (punctuation restoration with BERT)

I've found the script for punctuation restoration. And I have one question about this method. I will briefly explain the logic of the author. One of four tokens is assigned for each word: Other (0), ...
illuminato's user avatar
5 votes
2 answers
85 views

User profiling based on multiple posts

I currently have collected a dataset of different social media posts for each user with labels assigned to each user. I tried to use LSTM, and BERT for the text classification problem, So for each ...
Mac Sat's user avatar
  • 51
4 votes
1 answer
1k views

Bert for QuestionAnswering input exceeds 512

I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long ...
Simone's user avatar
  • 242
4 votes
1 answer
460 views

BERT word embedings for finding word definition

Can BERT, GPT or other contextualised embedings be used for finding word definitions? What would be the most effective and not complicated approach for tackling a sample task as described below. Map ...
piernik's user avatar
  • 51
4 votes
1 answer
388 views

Trained BERT models perform unpredictably on test set

We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label ...
PeterPaul's user avatar
4 votes
2 answers
8k views

Can we use BERT for only word embedding and then use SVM/RNN to do intent classification?

According to this article, "Systems used for intent classification contain the following two components: Word embedding, and a classifier." This article also evaluated BERT+SVM and Word2Vec+...
metk's user avatar
  • 41
4 votes
1 answer
1k views

NLP Transformers: How to get a fixed sentences embedding vectors size?

I'm loading a language model from torch hub (CamemBERT a French RoBERTa-based model) and using it do embed some sentences: ...
hzitoun's user avatar
  • 141
3 votes
2 answers
3k views

What are the elements in a BERT word embedding?

As far as I understand, BERT is a word embedding that can be fine-tuned or used directly. With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one ...
Emil's user avatar
  • 179
3 votes
1 answer
14k views

What is a 'hidden state' in BERT output?

I'm trying to understand the workings and output of BERT, and I'm wondering how/why each layer of BERT has a 'hidden state'. I understand what RNN's have a 'hidden state' that gets passed to each ...
Nick Koprowicz's user avatar
3 votes
2 answers
7k views

Similarity of words using BERTMODEL

I want to find the similarity of words using the BERT model within the NER task. I have my own dataset so, I don't want to use the pre-trained model. I do the following: ...
AFB's user avatar
  • 41
3 votes
1 answer
3k views

What is the difference between GPT blocks and BERT blocks

Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though. If GPT uses Decoder only and BERT uses ...
CD86's user avatar
  • 133
3 votes
2 answers
2k views

What is the difference between BERT architecture and vanilla Transformer architecture

I'm doing some research for the summarization task and found out BERT is derived from the Transformer model. In every blog about BERT that I have read, they focus on explaining what is a bidirectional ...
Luong Minh Tam's user avatar

1
2 3 4 5
8