Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
2
votes
1answer
4k views

How pre-trained BERT model generates word embeddings for out of vocabulary words?

Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
5
votes
1answer
4k views

How can I add custom numerical features for training to BERT fine tuning?

I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy. My original dataset consists of a pair of sentences (like MRPC ...
10
votes
2answers
6k views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a Word2Vec or Glove,...
10
votes
1answer
7k views

what is the first input to the decoder in a transformer model?

The image is from url: Jay Alammar on transformers K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each ...
17
votes
1answer
8k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
29
votes
4answers
22k views

What is purpose of the [CLS] token and why is its encoding output important?

I am reading this article on how to use BERT by Jay Alammar and I understand things up until: For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we ...
2
votes
1answer
3k views

How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>

As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode ...
5
votes
2answers
19k views

How to load the pre-trained BERT model from local/colab directory?

Hi i downloaded the BERT pretrained model (https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip) from here and saved to a directory in gogole colab and in local . when i ...
4
votes
1answer
3k views

What GPU size do I need to fine tune BERT base cased?

I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
3
votes
1answer
67 views

Trained BERT models perform unpredictably on test set

We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label ...
2
votes
3answers
1k views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
1
vote
1answer
1k views

What if My Word is not in Bert model vocabulary?

I am doing NER using Bert Model. I have encountered some words in my datasets which is not a part of bert vocabulary and i am getting the same error while converting words to ids. Can someone help me ...
0
votes
1answer
49 views

How can i get the vector of word using BERT?

I need to get word-vectors using BERT and got this function that i think it should be the one i need ...
0
votes
1answer
18 views

An issue for sub-word tokenization preprocessing transformer

I'm stacked with executing the sub-word tokenization preprocessing to use transformer. According to the tutorial on the article, I have executed the sample code. However, one function was not defined ...
0
votes
1answer
367 views

Is it possible to add new vocabulary to BERT's tokenizer when fine-tuning?

I want to fine-tune BERT by training it on a domain dataset of my own. The domain is specific and includes many terms that probably weren't included in the original dataset BERT was trained on. I know ...
0
votes
2answers
917 views

How to JUST represent words as embeddings by pretrained BERT?

I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding. After the ...