Questions tagged [bert]
BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers
26
questions
5
votes
1
answer
11k
views
How pre-trained BERT model generates word embeddings for out of vocabulary words?
Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
- 2,041
7
votes
1
answer
1k
views
Do transformers (e.g. BERT) have an unlimited input size?
There are various sources on the internet that claim that BERT has a fixed input size of 512 tokens (e.g. this, this, this, this ...). This magical number also appears in the BERT paper (Devlin et al. ...
- 193
21
votes
1
answer
11k
views
Can BERT do the next-word-predict task?
As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
- 691
14
votes
2
answers
8k
views
Preprocessing for Text Classification in Transformer Models (BERT variants)
This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models?
I remember for training a Word2Vec or Glove,...
- 4,207
6
votes
1
answer
7k
views
How can I add custom numerical features for training to BERT fine tuning?
I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy.
My original dataset consists of a pair of sentences (like MRPC ...
- 61
2
votes
1
answer
5k
views
What if My Word is not in Bert model vocabulary?
I am doing NER using Bert Model. I have encountered some words in my datasets which is not a part of bert vocabulary and i am getting the same error while converting words to ids. Can someone help me ...
- 23
0
votes
2
answers
4k
views
How to JUST represent words as embeddings by pretrained BERT?
I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding.
After the ...
- 137
67
votes
4
answers
78k
views
What is purpose of the [CLS] token and why is its encoding output important?
I am reading this article on how to use BERT by Jay Alammar and I understand things up until:
For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we ...
- 887
12
votes
1
answer
13k
views
what is the first input to the decoder in a transformer model?
The image is from url: Jay Alammar on transformers
K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each ...
- 574
8
votes
3
answers
5k
views
Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?
I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
- 415
6
votes
1
answer
9k
views
What GPU size do I need to fine tune BERT base cased?
I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
5
votes
2
answers
27k
views
How to load the pre-trained BERT model from local/colab directory?
Hi i downloaded the BERT pretrained model (https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip) from here and saved to a directory in gogole colab and in local .
when i ...
- 1,431
5
votes
3
answers
7k
views
where to store embeddings for similarity search?
I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
- 51
3
votes
2
answers
288
views
NER - What advantage does IO Format have over BIO Format
In this paper, the authors say that they used IO schema instead of BIO in their dataset, which, if I am not wrong, means they just tag the corresponding Entity Type or "O" in case the word ...
- 33
3
votes
1
answer
322
views
Trained BERT models perform unpredictably on test set
We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label ...
- 33
3
votes
1
answer
6k
views
BERT embedding layer
I am trying to figure how the embedding layer works for the pretrained BERT-base model. I am using pytorch and trying to dissect the following model:
...
- 163
2
votes
1
answer
5k
views
How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>
As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode ...
- 133
2
votes
1
answer
3k
views
Is it possible to add new vocabulary to BERT's tokenizer when fine-tuning?
I want to fine-tune BERT by training it on a domain dataset of my own. The domain is specific and includes many terms that probably weren't included in the original dataset BERT was trained on. I know ...
- 21
2
votes
2
answers
282
views
Usage of Word2Vec
Sorry for the basic doubt,
I would like to know if I can use my Word2Vec straight for classification without using LSTM. My assumption is it’s not possible because the ordering of the words will not ...
- 109
1
vote
1
answer
619
views
why multiple attention heads learn differently
In transformer architecture multi head attention blocks are used. While visualizing their output it can be seen that every layer has learnt different relations of words. e.g., layer 5 has learnt that &...
- 894
1
vote
1
answer
940
views
How to prepare texts to BERT/RoBERTa models?
I have an artificial corpus I've built (not a real language) where each document is composed of multiple sentences which again aren't really natural language sentences.
I want to train a language ...
- 141
1
vote
1
answer
2k
views
How can i get the vector of word using BERT?
I need to get word-vectors using BERT and got this function that i think it should be the one i need
...
- 113
1
vote
3
answers
2k
views
Fastest way for 1 vs all lookup on embeddings
I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array.
I wish to compare a new unseen text to all the 1 ...
- 173
0
votes
1
answer
321
views
Do I need to train a tokenizer when training SBERT with MLM?
I have trained a SBERT model with MLM on my own corpus which is somewhat domain specific using these guides:
https://ireneli.eu/2021/03/28/deep-learning-19-training-mlm-on-any-pre-trained-bert-models/
...
- 153
0
votes
0
answers
37
views
Mapping of an unseen Field/word to an existing description (in the input data), given Field and their respective descriptions as input/training data
I am working on a NLP problem.
Problem Statement
Given the input of fields & Labels and the respective descriptions, the goal is to the map a new unseen field to one of the most appropriate ...
- 319
0
votes
1
answer
102
views
An issue for sub-word tokenization preprocessing transformer
I'm stacked with executing the sub-word tokenization preprocessing to use transformer.
According to the tutorial on the article, I have executed the sample code.
However, one function was not defined ...
user107687