Questions tagged [bert]
BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers
357
questions
0
votes
2
answers
4k
views
How to JUST represent words as embeddings by pretrained BERT?
I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding.
After the ...
- 137
0
votes
1
answer
80
views
BERT Masked Language Model question
I have been reading about BERT from the internet, and from what I understand the point of masked language modelling for BERT pretraining is so that BERT will learn to guess a "masked" word ...
- 5
1
vote
1
answer
406
views
How to deal with class imbalance problem in natural language processing?
I am doing a NLP binary classification task, using Bert + softmax layer on top of it. The network uses cross-entropy loss.
When the ratio of positive class to negative class is 1:1 or 1:2, the model ...
- 145
0
votes
2
answers
2k
views
Is it good practice to use SMOTE when you have a data set that has imbalanced classes when using BERT model for text classification?
I had a question related to SMOTE. If you have a data set that is imbalanced, is it correct to use SMOTE when you are using BERT? I believe I read somewhere that you do not need to do this since BERT ...
- 133
1
vote
1
answer
272
views
dealing with HuggingFace's model's tokens
I have a few questions regarding tokenizing word/characters/emojis for different huggingface models.
From my understanding, a model would only perform best during inference if the token of the input ...
- 11
2
votes
1
answer
5k
views
What if My Word is not in Bert model vocabulary?
I am doing NER using Bert Model. I have encountered some words in my datasets which is not a part of bert vocabulary and i am getting the same error while converting words to ids. Can someone help me ...
- 23
0
votes
1
answer
28
views
Is LSTM or pretrained BERTForMasked LM usable for predicting changed word in a sentence using a small dataset? (2000 samples)
I have a small (2000 samples) dataset of newspaper headlines and their humorous conterparts where only one word is changed to sound silly, for example:
Original headline: Police <officer> ...
0
votes
1
answer
1k
views
Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?
I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it ...
- 2,041
0
votes
1
answer
365
views
Normalized 2D tensor values are not in range 0-1
Below function takes in 2D tensor and normalizes it using broadcasting .The issue is except all values to be in range 0-1 but the result has values outside this range . How to get all values in 2D ...
- 1,431
1
vote
0
answers
33
views
About Natural Question (NQ) benchmark in NLP [closed]
I recently learned that there is a benchmark called NQ.
https://ai.google.com/research/NaturalQuestions/visualization
Unlike other QA benchmarks which relevant document is povided with query, it has ...
- 11
0
votes
1
answer
3k
views
XLNET how to deal with text with more than 512 tokens?
From what I searched online, XLNET model is pre-trained with 512 tokens, and https://github.com/zihangdai/xlnet/issues/80 , I didn't find too much useful information on that either.
How does XLnet ...
- 113
1
vote
1
answer
2k
views
How to use BERT in seq2seq model? [closed]
I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need ...
- 113
3
votes
0
answers
180
views
Medical NER for French language
I'm currently exploring the options to extract medical NER specifically for French language. I tried SpaCy's general French NER but it wasn't helpful to the cause (...
- 285
1
vote
0
answers
27
views
Extracting layer output from Classification model of SimpleTransformer
I have fine tuned a bert base model for text classification task. Now, I want to extract hidden layer output so as to combine this output with other features to train a random forest model. Problem ...
- 21
1
vote
2
answers
501
views
Would there be any reason to pretrain BERT on specific texts?
So the official BERT English model is trained on Wikipedia and BookCurpos (source).
Now, for example, let's say I want to use BERT for Movies tag recommendation. Is there any reason for me to pretrain ...
- 265
1
vote
1
answer
735
views
Are all 110 million parameter in bert are trainable
I am trying to understand are all these 110 million parameters trainable of bert uncased model. Is there any non trainable parameters in this image below?
By trainable I understand they are ...
- 155
3
votes
1
answer
5k
views
Backpropagation of a transformer
when a transformer model is trained there is linear layer in the end of decoder which i understand is a fully connected neural network. During training of a transformer model when a loss is obtained ...
- 155
1
vote
1
answer
405
views
what's the motivation behind BERT masking 2 words in a sentence?
bert and the more recent t5 ablation study, agree that
using a denoising objective always results in better downstream task performance compared to a language model
where denoising == masked-lm == ...
- 1,357
1
vote
1
answer
1k
views
Bert and SVM classification
I'm trying to understand the concepts in the title and how they fit into the task of binary classification. According to my understanding so far, you can encode text using various feature-extraction ...
- 111
0
votes
1
answer
309
views
Why does BERT embedding increase the number of tokens?
I am new to DataScience and trying to implement BERT embedding for one of my problems. But I am having one doubt here.
I am trying to embed the following sentence with BERT - "Twinkle twinkle ...
2
votes
1
answer
3k
views
What is the difference between GPT blocks and BERT blocks
Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though.
If GPT uses Decoder only and BERT uses ...
- 123
0
votes
1
answer
644
views
What is the number of neurons for the input layer of the BERT?
I think it is the vocab size. However I am not sure and I appreciate your help.
- 117
3
votes
2
answers
1k
views
Detecting grammatical errors with BERT
We fine-tuned BERT (bert-base-uncased) model with CoLA dataset for sentence classification task. The dataset is a mix of ...
- 285
5
votes
1
answer
2k
views
Is a BiLSTM layer required if we use BERT?
I am new to Deep learning based NLP and I have a doubt -
I am trying to build a NER model and I found some journals where people are relying on BERT-BiLSTM-CRF model for it. As far as I know BERT is a ...
1
vote
1
answer
133
views
Inference order in BERT masking task
In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order?
For example:
The ...
1
vote
0
answers
526
views
How do I handle class imbalance for text data when using pretrained models like BERT?
I have a skewed dataset consisting of samples of the form:
Category 1 10000
Category 2 2000
Category 3 400
Category 4 300
Category 5 100
The dataset ...
- 11
5
votes
2
answers
2k
views
how to run bert's pretrained model word embeddings faster?
I'm trying to get word embeddings for clinical data using microsoft/pubmedbert.
I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, ...
- 138
0
votes
3
answers
4k
views
BERT :dropout(): argument 'input' (position 1) must be Tensor, not str
I am new to NLP and would like to build a BERT model for sentiment analysis so I am following this tutorial.
However, I am getting the error below:
...
- 137
0
votes
1
answer
192
views
Which script can be used to finetune BERT for SQuAD question answering in Hugging Face library?
I have gone through lot of blogs which talk about run_squad.py script from Hugging Face, but I could not find it in the latest repo. So, which script has to be used ...
- 113
0
votes
1
answer
2k
views
BERT uses WordPiece, RoBERTa uses BPE
In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned:
The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
- 117
3
votes
1
answer
322
views
Trained BERT models perform unpredictably on test set
We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label ...
- 33
0
votes
0
answers
148
views
BERT for classification model degenerates into all-positive predictions
As a learning project, I'm training a BERT model with the CoLA dataset to detect sentence acceptability. Unfortunately my model is learning to classify every instance as "acceptable", and I'...
- 101
3
votes
2
answers
2k
views
What is the difference between BERT architecture and vanilla Transformer architecture
I'm doing some research for the summarization task and found out BERT is derived from the Transformer model. In every blog about BERT that I have read, they focus on explaining what is a bidirectional ...
- 133
0
votes
1
answer
3k
views
BERT minimal batch size
Is there a minimum batch size for training/re-fining a BERT model on custom data?
Could you name any cases where a mini batch size between 1-8 would make sense?
Would a batch size of 1 make sense at ...
- 316
0
votes
0
answers
205
views
BERT data cleaning [duplicate]
I am wondering which data cleaning steps should be performed if you want to re-fine a BERT model on custom text data.
Which steps should be performed?
Does it make sense to perform a stemming or ...
- 316
0
votes
1
answer
92
views
Using BERT for the first time, what are the two columns of my test_results.tsv?
I followed the steps to feed in both dev, test, train.tsv to the model, trained it, then tried to classify test data, and I only have 1 feature, and the classification is binary, 1 or 0. I assumed my ...
- 1
5
votes
1
answer
11k
views
How pre-trained BERT model generates word embeddings for out of vocabulary words?
Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
- 2,041
2
votes
1
answer
404
views
From where does BERT get the tokens it predicts?
When BERT is used for masked language modeling, it masks a token and then tries to predict it.
What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression ...
- 213
1
vote
1
answer
455
views
Can I fine-tune the BERT on a dissimilar/unrelated task?
In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned:
"During pre-training, the model is trained on unlabeled data over different pre-training tasks."
I am not sure if I ...
- 117
1
vote
0
answers
214
views
NLP Bert model to to calculate text similarity, same sentence but not close similarity
Dear expert here:
I have a simple program to calculate text similarity. The program is copied from internet. Initially, I have a list of sentences or stored in db and fetched from db, then
I make the ...
- 183
1
vote
0
answers
154
views
How to apply pruning on a BERT model?
I have trained a BERT model using ktrain (tensorflow wrapper) to recognize emotion on text, it works but it suffers from really slow inference. That makes my model not suitable for a production ...
0
votes
1
answer
2k
views
Using pretrained LSTM and Bert Models in CPU Only Environment - How to speed up Predictions?
I have trained two text classification models using GPU on Azure. The models are the following
Bert (ktrain)
Lstm Word2Vec (tensorflow)
Exaples of the code can be found here: nlp
I saved the models ...
1
vote
1
answer
868
views
If i use use BERT embeddings for if cosine(sent1,sent2) > 0.9, then is it fair to assume s1 and s2 are similar
According to BERT author Jacob Devlin: I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is doing average pooling over the word tokens ...
- 113
0
votes
1
answer
745
views
Is it possible to predict sentiment of unlabelled dataset using BERT?
I have a large unlabeled dataset and I want to predict sentiment for each document in this dataset. I want to know, is it possible that I can use BERT for sentiment analysis of unlabeled data? I have ...
- 165
-3
votes
1
answer
174
views
Generative chatbots with BERT pretrained vectors
Most places seem to train generative chatbots with one hot encoded vectors. See here for example, and even the official tutorial on pytorch.
But using one hot encoded vectors are undoubtedly the worst ...
- 97
1
vote
1
answer
984
views
What is the typical accuracy of masked language models during BERT pretraining?
I was reading the BERT paper but I didn't find any tables concerning the performance of the masked language models during pretraining. Does anyone know the accuracy of BERT's masked language model?
- 13
4
votes
1
answer
980
views
Bert for QuestionAnswering input exceeds 512
I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long ...
- 242
11
votes
2
answers
12k
views
Does BERT has any advantage over GPT3?
I have read a couple of documents that explain in detail about the greater edge that GPT-3(Generative Pre-trained Transformer-3) has over BERT(Bidirectional Encoder Representation from Transformers). ...
- 213
2
votes
1
answer
321
views
Question about BERT embeddings with high cosine similarity
Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?
- 23
2
votes
2
answers
2k
views
Does finetuning BERT involving updating all of the parameters or just the final classification layer?
Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP. But during finetuning, for example trying ...
- 21