Questions tagged [bert]
BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers
357
questions
1
vote
1
answer
268
views
BERT is running out of memory in forward pass for my dictionary
Running code from this answer, my BERT is running out for my 4k words dictionary.
I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly:
...
CommunityBot
- 1
0
votes
0
answers
23
views
Assign layers and weights in BERT
I print the weight names and shape of the BERT transformer. Now, I want to assign the printed weight to the layers in the transformers architecture:
In the following, I can assign query, key and ...
- 101
1
vote
1
answer
1k
views
Bert and SVM classification
I'm trying to understand the concepts in the title and how they fit into the task of binary classification. According to my understanding so far, you can encode text using various feature-extraction ...
CommunityBot
- 1
0
votes
0
answers
14
views
What happens when I set is_decoder to True in the bert API from huggingface?
Please help me understand the implications of initialising the bert model from huggingface with is_decoder parameter set to True
...
- 821
1
vote
1
answer
50
views
How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder
Bert has two types of tasks that it uses to learn contextual word embeddings:
Masked word prediction
Next sentence prediction
I have read the paper and even there the training details are a little ...
- 821
1
vote
1
answer
191
views
Is there any concern for a pretrained model to overfitting to a fine-tuning task that has overlapping pretraining and training data?
Let's say my language model is pretrained on a general text corpus, and I want to use it for some specific downstream task that has it's datasets also included in the general corpus, is there any ...
CommunityBot
- 1
0
votes
0
answers
10
views
F1 and Exact-Match (EM) Score in Extractive QA NLP
I have a question as to how the F1 should be calculated in NLP and whether the text normalization is optional or not.
So I have been working on a project where we created a closed-domain extractive QA ...
- 101
1
vote
1
answer
359
views
Bertopic with embedding: unable to use find_topic
I've used BERTopic with success for the following tasks: get topics, visualise (topics, barcharts, documents ...) and DTM (extended to get area plot with considerable success).
However, I am unable to ...
CommunityBot
- 1
0
votes
2
answers
1k
views
Get sentence embeddings of transformer-based models
I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...).
I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-...
CommunityBot
- 1
0
votes
1
answer
135
views
Doubt in ELMO, BERT, Word2Vec
I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
CommunityBot
- 1
0
votes
1
answer
92
views
Using BERT for the first time, what are the two columns of my test_results.tsv?
I followed the steps to feed in both dev, test, train.tsv to the model, trained it, then tried to classify test data, and I only have 1 feature, and the classification is binary, 1 or 0. I assumed my ...
CommunityBot
- 1
0
votes
1
answer
1k
views
Using KerasClassifier for training neural network
I created a simple neural network for binary spam/ham text classification using pretrained BERT transformer. The current pure-keras implementation works fine. I wanted however to plot certain metrics ...
CommunityBot
- 1
0
votes
1
answer
315
views
How to write a generator to fine-tune transformer based models (Tensorflow)
I have been trying to write a generator for DistillBertFast model
...
CommunityBot
- 1
1
vote
2
answers
162
views
Using BERT to extract a list of words and phrases from documents
I have a list of words and phrases (~3k items). What are my options to extract them from documents (~3M of job descriptions) with NLP? I do not have labeled data.
For example my list of words and ...
CommunityBot
- 1
0
votes
0
answers
16
views
TFRobertaSequenceClassification for Address Normalization task
I have dataset with two column: one with faulty addresses, and other with correct addresses. I want to train a model such that, I can use it later for correcting all the incoming faulty addresses.
I ...
0
votes
0
answers
33
views
How can I avoid the irrelevant number of sentences in the result?
The nature of the data I have is not arranged; however, I'm trying to extract the appropriate sentences for each query as a sample for ground truth. Also, the most critical problem is that I use the ...
- 17
0
votes
0
answers
30
views
How to use Bertweet model for topic modeling
The problem is implementation of Bertweet in a topic-modeling project with understandable output like BERTopic, i want to use it on a relatively large (20k tweets) unlabelled dataset to segment it ...
- 1
1
vote
2
answers
88
views
Training model using BERT
I have generated dataset using chat gpt. Dataset has 9000 data recodes. It's 6 class sentiment analysis. classes are 0,1,2,3,4,5
I used 3000 recodes for training, 1200 recods for validation and ...
- 266
0
votes
1
answer
140
views
When would you use word2vec over BERT?
I am very new to Machine Learning and I have recently been exposed to word2vec and BERT.
From what I know, word2vec provides a vector representation of words, but is limited to its dictionary ...
CommunityBot
- 1
1
vote
1
answer
113
views
Does high number of output labels affect the performance of BERT and how to handle the class imbalance issue while doing multi text classification?
I am using BERT to do multiclass text classification. The number of output classes I have to predict from is: 116 and there is high degree of class imbalance that I see.
We have the following kind of ...
CommunityBot
- 1
1
vote
2
answers
740
views
Fine-tuned MLM based RoBERTa not improving performance
We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM.
We took some sample data points (2M+) & fine-tuned ...
CommunityBot
- 1
0
votes
1
answer
303
views
Loading a Model with weights and optimizers without creating an instance in PyTorch
I recently downloaded Camembert Model to fine-tune it for my purpose.
Upon unzipping the file the contents are:
Upon loading the model.pt file using pytorch:
...
CommunityBot
- 1
2
votes
1
answer
2k
views
HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00
I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead ...
CommunityBot
- 1
0
votes
1
answer
137
views
Using BERT for co-reference resolving, what's the loss function?
I'm working my way around using BERT for co-reference resolving. I'm following this highly-cited paper BERT for Coreference Resolution: Baselines and Analysis (https://arxiv.org/pdf/1908.09091.pdf). I ...
CommunityBot
- 1
7
votes
1
answer
9k
views
How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?
I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
CommunityBot
- 1
0
votes
0
answers
39
views
Help understanding working of KeyBERT for keyphrase extraction
I'm fairly new to reading and understanding research papers, so I wanted to get a second opinion on whether my understanding of KeyBERT was correct. Here is a high level overview of my understanding ...
- 33
2
votes
1
answer
359
views
Predicting word from a set of words
My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
CommunityBot
- 1
0
votes
0
answers
28
views
Combining Textual, Categorical and Numerical data for Semantic Search using SentenceTransformers model
I'm building a food semantic search model and I want to use a pre-trained SentenceTransformers model with cosine similarity. I'm using Epicurious dataset for the corpus which consists of textual (&...
- 1
0
votes
0
answers
22
views
How to deal with short text data using NLP models?
Now I want to use my own domain data to train NLP model like BERT. The following is the details of my data:
data length distribution: over 70% of my data has the length shorter than 5 and the largest ...
1
vote
1
answer
979
views
Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)
I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:
...
CommunityBot
- 1
0
votes
2
answers
32
views
building embeddings for Phrases from scratch
I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
- 11
1
vote
1
answer
285
views
BERTopic Visualization
I new to topic modeling and I'm trying to use BERTopic inside of PyCharm. I'm struggling to ...
CommunityBot
- 1
0
votes
2
answers
65
views
Fine-tune GPT on sketch data (stroke-3)
These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it.
I was planning on using some pretrained models that are ...
CommunityBot
- 1
0
votes
2
answers
147
views
How to improve the evaluation score for highly imbalanced dataset?
I have trained my BERT model(bert-base-cased) to detect toxic comments. I used the Toxic Comment Classification Challenge dataset from the Kaggle. My accuracy is 98% and the AUROC for various sub-...
CommunityBot
- 1
0
votes
1
answer
198
views
SBERT Embeddings from Conversations
I have a dataset consisting of text-based conversations between two humans. One conversation has on average 20 turns and can look as follows:
...
CommunityBot
- 1
1
vote
1
answer
384
views
How does BERT work for Aspect-Based sentiment analysis?
I have recently used a package to perform Aspect-Based Sentiment Analysis (ABSA) through a BERT model.
Briefly, the model takes two inputs:
words that constitute the aspects
a sentence on which we ...
CommunityBot
- 1
3
votes
1
answer
2k
views
How Transformer is Bidirectional - Machine Learning
Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/...
CommunityBot
- 1
1
vote
2
answers
713
views
BERT classifier with Ktrain API is unable to predict new data
I have trained a classifier for sentiment analysis using BERT architecture.
I am able to train the classifier and I am getting a validation accuracy of 87%. But whenever I feed in test data, or some ...
CommunityBot
- 1
0
votes
1
answer
275
views
how to improve my imbalanced data NLP model?
I want to classify a patient's health as a prediction probability and get the top 10 most ill patients in a hospital. I have patient's condition notes, medical notes, diagnoses notes, and lab notes ...
CommunityBot
- 1
1
vote
1
answer
3k
views
Overfitting in Huggingface's TFBertForSequenceClassification
I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
CommunityBot
- 1
0
votes
1
answer
442
views
Is there bias in matrix multiplications for self attention
When the query matrix Q is computed as $XW_Q$, ($W_Q$ is the weight matrix for the queries), is it implemented as a linear layer without bias? I see some blogs saying there is are bias terms as well.
...
CommunityBot
- 1
0
votes
1
answer
1k
views
Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?
I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it ...
CommunityBot
- 1
3
votes
2
answers
1k
views
Detecting grammatical errors with BERT
We fine-tuned BERT (bert-base-uncased) model with CoLA dataset for sentence classification task. The dataset is a mix of ...
CommunityBot
- 1
2
votes
2
answers
549
views
How to get sentiment score for a word in a given dataset
I have a sentiment analysis dataset that is labeled in three categories: positive, negative, and neutral. I also have a list of words (mostly nouns), for which I want to calculate the sentiment value, ...
CommunityBot
- 1
2
votes
1
answer
936
views
How to classify neutral sentiments using BERT
We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also?
https://colab.research.google.com/github/google-...
CommunityBot
- 1
4
votes
2
answers
3k
views
Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?
It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving ...
- 23.8k
0
votes
0
answers
280
views
Adapting a BERT-based model from HuggingFace for NER (named entity recognition) and RE (relation extraction)?
Context: NER (named entity recognition) and RE (relation extraction) from sentences obtained from radiology reports (medical text).
There is a BERT-based model from HuggingFace I would like to use for ...
- 377
0
votes
0
answers
32
views
Classification errors on 'bert-base-uncased' text classifier
Disclaimer : This is a long question, please be patient. Thanks in advance
I am using bert-base-uncased for text-classification. I have 11 classes, and the classification is happening alright for most ...
0
votes
0
answers
28
views
BERT + tensorflow + deterministic
Im using BERT in tensorflow, but when I try to turn it deterministic I got the error: "When determinism is enabled, random ops must have a seed specified.
[[{{node dropout/dropout/random_uniform/...
0
votes
0
answers
13
views
How to analyze social media data to see its impact on a game's sales
I work for a console gaming giant. We forecasted the sales for a RPG game that was to be released few months back. But the actual sales was twice the forecast. This compelled the developers to ...
