Questions tagged [bert]
BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers
352
questions
0
votes
0
answers
15
views
How to deal with short text data using NLP models?
Now I want to use my own domain data to train NLP model like BERT. The following is the details of my data:
data length distribution: over 70% of my data has the length shorter than 5 and the largest ...
0
votes
0
answers
129
views
Adapting a BERT-based model from HuggingFace for NER (named entity recognition) and RE (relation extraction)?
Context: NER (named entity recognition) and RE (relation extraction) from sentences obtained from radiology reports (medical text).
There is a BERT-based model from HuggingFace I would like to use for ...
- 377
2
votes
2
answers
1k
views
Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?
It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving ...
- 123
0
votes
0
answers
26
views
Classification errors on 'bert-base-uncased' text classifier
Disclaimer : This is a long question, please be patient. Thanks in advance
I am using bert-base-uncased for text-classification. I have 11 classes, and the classification is happening alright for most ...
0
votes
0
answers
21
views
BERT + tensorflow + deterministic
Im using BERT in tensorflow, but when I try to turn it deterministic I got the error: "When determinism is enabled, random ops must have a seed specified.
[[{{node dropout/dropout/random_uniform/...
0
votes
0
answers
12
views
How to analyze social media data to see its impact on a game's sales
I work for a console gaming giant. We forecasted the sales for a RPG game that was to be released few months back. But the actual sales was twice the forecast. This compelled the developers to ...
0
votes
2
answers
27
views
building embeddings for Phrases from scratch
I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
0
votes
1
answer
30
views
Word embeddings
I m looking into word embedding and I would like to ask if I could train words or sentences in two layers. And if I wanted that one layer is more important, how could I calculate it? For example ...
0
votes
1
answer
14
views
BERT is a supervised learning or semi-supervised learning?
I use 'bert-base-cased' pre-trained model for encoding a dataset of text that was labeled to labels 0, 1. Then the encoded dataset is trained using BERT model imported from Transformer library. Does ...
- 3
0
votes
1
answer
53
views
Can bert uncased predict text classification on foreign data?
I am trying to do the fake news/real news classification and used a pre-trained bert uncased model as transfer learning and it gave a solid 81% accuracy. But the problem is while doing sanity checks, ...
0
votes
0
answers
10
views
Use unclassified texts to improve BERT and RNN GRUN token classification model
I have a training (gold) labelled dataset than consists of 10000 sentences. The task is to create a model that classifies correctly unseen data with B-I-O tags.
I have used a BERT and a GRU RNN model.
...
0
votes
1
answer
38
views
Building BERT tokenizer with custom data
I'm wondering if there is a way to train our own Bert tokenizer instead of using pre-trained tokenizer provided by huggingface?
0
votes
1
answer
27
views
what is the difference between NSP and text prediction
In BERT, NSP (Next Sentence Prediction) is for predicting next sentence based on context and Text prediction task is also for predicting next word or phrases.
So, both are for predicting next sentence ...
- 37
1
vote
0
answers
42
views
I can't get good performance from BERT
I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin
I compare an LSTM model with a glove model and a BERT model. I ...
- 131
0
votes
0
answers
21
views
Why type of help can we get on stack Exchange: Datascience?
I have a text classification dataset. The aim is to predict the category of an article based on its title. I have about 100 categories and 10 thousands instances. I've tried models like RNN, LSTM. I'...
- 131
1
vote
0
answers
85
views
Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding
I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
- 113
0
votes
2
answers
56
views
detect/generate possible tokens for the dataset (name,type/category,signatures)
I have a dataset in the following format:
name, type, signature
Eg1 : A, 2, abc123
Eg2 : A, 2, ab3
Eg3 : A, 2, addc1
If we need to train the following dataset ...
- 53
0
votes
0
answers
20
views
Huggingface not saving model checkpoint
I am trying to train T5 model. This is how my training arguments look like:
...
- 103
0
votes
1
answer
39
views
Bert model for document sentiment classification
I am trying to fine-tune a Bert model for sentiment analysis. Instead of one sentence, my inputs are documents (including several sentences) and I am not removing dots. I was wondering if is it okay ...
0
votes
0
answers
54
views
Clustering with BERT. Why are my clusters overlapped? How to improve BERT embeddings?
I am trying to create BERT embeddings of text data, then use dimensionality reduction and cluster. I tried with some big datasets like amazon reviews and 20newsgroups, but whenever I created ...
0
votes
0
answers
33
views
The embedding output of bert
I want to get the embedding matrix of the Bert model (the input before the first block layers) to feed it into another architecture. I really appreciate it if you help me with that. Thanks
1
vote
2
answers
506
views
Fine-tuned MLM based RoBERTa not improving performance
We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM.
We took some sample data points (2M+) & fine-tuned ...
- 11
1
vote
1
answer
47
views
0
votes
0
answers
12
views
Unstable training of BERT binary sequence classification. Higher loss but lower gradients
I'm training a BERT sequence classifier on a custom dataset. When the training starts, the loss is at around ~0.4 in a few steps. I print the absolute sum of gradients for each layer/item in the model ...
- 1
0
votes
0
answers
17
views
Creating Word Embeddings using BERT for Machine-Generated Text Data
I have a dataset of machine-generated sequences that are not natural language, but the order of the words in the sequence is important. I want to create word embeddings using BERT to capture the ...
- 101
0
votes
1
answer
196
views
BERTopic: Is it okay to ignore the first two topics?
I used BERTopic to generate a topic model over a large dataset of texts. The result is very appealing and the modeled topics are mostly perfectly interpretable for a human, especially compared to ...
- 1
0
votes
1
answer
93
views
Doubt in ELMO, BERT, Word2Vec
I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
0
votes
2
answers
63
views
Fine-tune GPT on sketch data (stroke-3)
These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it.
I was planning on using some pretrained models that are ...
- 31
4
votes
1
answer
868
views
Do transformers (e.g. BERT) have an unlimited input size?
There are various sources on the internet that claim that BERT has a fixed input size of 512 tokens (e.g. this, this, this, this ...). This magical number also appears in the BERT paper (Devlin et al. ...
- 163
0
votes
0
answers
11
views
How to effectively use CharBERT for text similarity?
I'm looking into CharBERT for an university project, and I noticed that it was finetuned on many tasks like sentiment analysis, NER, and so on. I tried to use it to do text similarity by using only ...
- 1
0
votes
0
answers
10
views
How can we integrate zero shot classification with a supervised classifier?
We have two sets of labels that are known and unknown to the supervised classifier (SC).
We infer for a test example using the SC and a zero shot classifier (ZC).
Let's assume, our inference datapoint ...
1
vote
2
answers
106
views
Using BERT to extract a list of words and phrases from documents
I have a list of words and phrases (~3k items). What are my options to extract them from documents (~3M of job descriptions) with NLP? I do not have labeled data.
For example my list of words and ...
- 405
1
vote
1
answer
57
views
Does high number of output labels affect the performance of BERT and how to handle the class imbalance issue while doing multi text classification?
I am using BERT to do multiclass text classification. The number of output classes I have to predict from is: 116 and there is high degree of class imbalance that I see.
We have the following kind of ...
0
votes
0
answers
80
views
Is there any concern for a pretrained model to overfitting to a fine-tuning task that has overlapping pretraining and training data?
Let's say my language model is pretrained on a general text corpus, and I want to use it for some specific downstream task that has it's datasets also included in the general corpus, is there any ...
- 1
0
votes
0
answers
7
views
Which is the best choice for evaluating models on small and unbalanced textual datasets?
We are dealing with a small multilabel dataset (around 15k samples) of texts that is imbalanced. Some classes have more than 4k samples and others have around 700 samples. We are using a classifier ...
- 139
1
vote
1
answer
43
views
Is it valid changing the classification treshold of neural networks for improving the classification performance?
I'm dealing with text classification using BERT pre-trained model with a multiclass imbalanced dataset.
When we use a 0.5 default classification threshold we obtain a f1 measure of around 0.7.
But we ...
- 139
0
votes
1
answer
139
views
SBERT Embeddings from Conversations
I have a dataset consisting of text-based conversations between two humans. One conversation has on average 20 turns and can look as follows:
...
- 121
1
vote
1
answer
226
views
How does BERT work for Aspect-Based sentiment analysis?
I have recently used a package to perform Aspect-Based Sentiment Analysis (ABSA) through a BERT model.
Briefly, the model takes two inputs:
words that constitute the aspects
a sentence on which we ...
0
votes
0
answers
12
views
How do I predict the BERT embedding of a vector that is associated with the given X?
I have a list of sentences called X. Each sentence is associated with a sentence in a list Y. I'm trying to use BERT to create a model that, given a sentence in X, will predict which of the sentences ...
- 149
1
vote
1
answer
57
views
What to do if my dataset have only One instance for class in classification?
I am working on a benchmark dataset for text classification. The dataset has about 300 classes, and approximately 50 of these classes have a single instance. In a paper that used fine-tuning BERT, the ...
0
votes
0
answers
138
views
How do I improve the accuracy of a BERT-based multilabel text classification model?
I have a database table with 79,512 rows, each of which describes a category. Each row has a title and a description, and can even have a supercategory. Often, supercategories have categories.
I'm ...
- 149
0
votes
1
answer
171
views
Combining text and image features with different scales
I have computed text features using [SBERT][1] and image features using VGG-16. The text features range from -1.58 to 1.58, whereas the image features range between 0 and 521. I would want to ...
- 1
0
votes
1
answer
86
views
Sentiment analysis BERT vs Model from scratch
I am working on building a sentiment analyzer, the data I would like to analyze is social media data from twitter, once I have created a the model I want to integrate it into a simply webpage.
I have ...
0
votes
2
answers
123
views
how can I translate Whisper encodings to SBERT embeddings?
I'm using the Whisper model to recognize speech, and then matching the output text against a list of known questions by generating SBERT embeddings from the text and ranking the known questions by ...
- 103
0
votes
1
answer
67
views
why some authors said that BERT cannot be used for text prediction?
I was trying to get a grasp about BERT and found this post in DS StackExchange:
Can BERT do the next-word-predict task?
In broad terms, it says that Bert cannot be used for next-word prediction. I ...
- 217
0
votes
0
answers
40
views
FinBERT out of the box performance testing
I'm trying to perform an out of the box performance test for FinBERT using the financialphrasebank dataset(sentiment analysis) to get a baseline performance before I start finetuning the model. The ...
- 1
0
votes
0
answers
29
views
Pruning using BERTology
I am trying out some BERT based models for a question and answering task. I need models trained on squad v2.0. To cut down on the inference time , I'm trying out pruning. I came across the BERTology ...
- 103
0
votes
0
answers
101
views
cannot freeze RoBERTa model base layer
I want to Freeze my RoBERTa model base layer and only train on my classification layer, but i get the following error 'TFRobertaEmbeddings' object has no attribute 'parameters'. Here is my code
...
- 1
1
vote
1
answer
215
views
BERTopic Visualization
I new to topic modeling and I'm trying to use BERTopic inside of PyCharm. I'm struggling to ...
- 141
0
votes
0
answers
45
views
Predicting same tokens as base BERT model for token classification on custom dataset
I have a custom dataset with custom tag for each token in the text. I want to train a BERT model for classifying each token into its corresponding category. To do ...
user144851