Questions tagged [bert]
BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers
357
questions
0
votes
1
answer
275
views
how to improve my imbalanced data NLP model?
I want to classify a patient's health as a prediction probability and get the top 10 most ill patients in a hospital. I have patient's condition notes, medical notes, diagnoses notes, and lab notes ...
- 138
1
vote
1
answer
39
views
Combining models trained on a multilingual multi-source corpus
Consider the following training corpora:
dataset1: composed of French instances
dataset2: dataset1 + Arabic instances
test_dataset (for both scenarios): composed of French instances
(the same ...
- 75
1
vote
1
answer
480
views
why is the BERT NSP task useful for sentence classification tasks?
BERT pre-trains the special [CLS] token on the NSP task - for every pair A-B predicting whether sentence B follows sentence A in ...
- 1,357
1
vote
0
answers
215
views
Using Sentence-Bert with other features in scikit-learn
I have a dataset where one feature is text and 4 more features. Sentence-Bert vectorizer transforms text data into tensors. I can use these sparse matrices directly with a machine learning classifier. ...
- 11
1
vote
0
answers
282
views
NER prections with distilbert transformer model
I am trying to extract 'agreement date' label from a corpus of legal contracts. In the train dataset, I used pytorch-transformer model to train.
...
- 56
0
votes
0
answers
400
views
BERT Text Classification Model gives error
I am fine-tuning a BERT Model for text classification with Tensorflow. Here is my code for building the model:
...
- 129
1
vote
1
answer
754
views
Is it possible to fine-tuning BERT by training it on multiple datasets? (Each dataset having it's own purpose)
BERT can be fine-tuned on a dataset for a specific task. Is it possible to fine-tune it on all these datasets for different tasks and then be utilized for these tasks instead of fine-tuning a BERT ...
0
votes
0
answers
152
views
How long does it take to fine-tune XLNet?
XLNet takes a lot more time than BERT during pre-training. This results in XLNet performing better than BERT in over 20 NLP tasks. How long does XLNet take for fine-tuning (let's assume this is ...
1
vote
0
answers
345
views
Is it possible to fine-tune a (Spanish RoBERTa) model for a different task?
I'm doing sentiment analysis of Spanish tweets.
After reviewing some of the recent literature, I've seen that there's been a most recent effort to train a RoBERTa model exclusively on Spanish text. It ...
- 131
3
votes
1
answer
2k
views
Does BERT need supervised data only when fine-tuning?
I've read many articles and papers mentioning how unsupervised training is conducted while pre-training a BERT model. I would like to know if it is possible to fine-tune a BERT model in an ...
2
votes
2
answers
549
views
How to get sentiment score for a word in a given dataset
I have a sentiment analysis dataset that is labeled in three categories: positive, negative, and neutral. I also have a list of words (mostly nouns), for which I want to calculate the sentiment value, ...
- 21
1
vote
1
answer
182
views
For text classification, would a BoW or Word Embeddings based model ever be better than a Language Model?
I've done a bit of research, with this being the best as far as objectively measuring quality, but wanted to ask from a theoretical perspective if BoW-based models (e.g. using TF-IDF) or word ...
- 111
1
vote
1
answer
39
views
How BERT model differentiate words with different meanings? [closed]
How BERT model differentiate between words with different meanings e.g. #Transformers like name of the movie or name of a library by @huggingface?
- 263
1
vote
1
answer
370
views
Annotating NER dataset
I am working on annotating a dataset for the purpose of named entity recognition.
In principle, I have seen that for multi-phrase (not single word) elements, annotations work like this (see this ...
- 166
2
votes
1
answer
3k
views
Is it possible to add new vocabulary to BERT's tokenizer when fine-tuning?
I want to fine-tune BERT by training it on a domain dataset of my own. The domain is specific and includes many terms that probably weren't included in the original dataset BERT was trained on. I know ...
- 21
0
votes
2
answers
147
views
How to improve the evaluation score for highly imbalanced dataset?
I have trained my BERT model(bert-base-cased) to detect toxic comments. I used the Toxic Comment Classification Challenge dataset from the Kaggle. My accuracy is 98% and the AUROC for various sub-...
0
votes
0
answers
529
views
Can I average the BERT embeddings of multiple instances of the same word to get one vector representation of the word?
In the project I'm working on right now I would like to get one embedding for every unique lemma in a corpus. Could I get this by averaging the embeddings of every instance of a lemma?
For example, ...
- 1
0
votes
1
answer
59
views
Remedy for small batch size?
I am trying to reproduce results of other people's research, but we cannot afford to do it with the same batch size as theirs, due to limited computing resources.
The method they use is a simple ...
- 3
1
vote
0
answers
41
views
How to do Bert Finetuning of failure cases?
I have a large dataset(public available) of text that is labelled. However the test distribution (actual production setting of company) while similar is not from the same source and thus tends to fail ...
- 73
2
votes
1
answer
2k
views
HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00
I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead ...
- 21
0
votes
1
answer
102
views
An issue for sub-word tokenization preprocessing transformer
I'm stacked with executing the sub-word tokenization preprocessing to use transformer.
According to the tutorial on the article, I have executed the sample code.
However, one function was not defined ...
user107687
2
votes
2
answers
1k
views
Transformers (BERT) vs LSTM on Sentiment Analysis/NER - dataset sizes comparison
I am aware (continuously learning) of the advantages of Transformers over LSTMs.
At the same time, I was wondering from the viewpoint of size of the data needed, contrast of those two techniques, ...
- 166
0
votes
1
answer
486
views
Error to load a pre-trained BERT model
Background
I'm reading this article about a natural language task, named entity recognition and trying to load a pre-trained BERT model on Google colaboratory.
How can I fix an error to load a pre-...
user107687
0
votes
1
answer
43
views
How to interepret BERT Attention
Can we tell BERT extracts local features?
For example consider the sentence "This is my first sentence. This is my second sentence".
Now How Bert extracts the features.
attention is computed ...
- 239
0
votes
1
answer
3k
views
Bert to extract local features
Bert is pre-trained model which can be fine-tuned for the text classification. How to extract local features using BERT
- 239
3
votes
0
answers
1k
views
Combining heterogeneous numerical and text features
We want to solve a regression problem of the form "given two objects $x$ and $y$, predict their score (think about it as a similarity) $w(x,y)$". We have 2 types of features:
For each ...
- 81
1
vote
1
answer
105
views
BERT Optimization for Production
I'm using BERT to transform text into 768 dim vector, It's multilingual :
...
- 45
1
vote
1
answer
120
views
Document ranking on a web scraped dataset without any labelled data
I want to create a document ranking model which returns similar rows in the dataset for a sample query. The text in this corpus is standard english but without any labels (ie no query-related ...
- 11
8
votes
2
answers
16k
views
What is the difference between BERT and Roberta
I want to understand the difference between BERT and Roberta. I saw the article below.
https://towardsdatascience.com/bert-roberta-distilbert-xlnet-which-one-to-use-3d5ab82ba5f8
It mentions that ...
- 183
1
vote
0
answers
43
views
NLP-Problem, language model BERT?
Right now I am in the process of deciding on my masters thesis topic. Right now I and my professor are thinking about the possibility of having a large dataset of product requirements given in a ...
- 11
0
votes
1
answer
315
views
How to write a generator to fine-tune transformer based models (Tensorflow)
I have been trying to write a generator for DistillBertFast model
...
- 231
0
votes
1
answer
24
views
Influence of label names on the classfierier perfromance
I am building a text classifier, the labels in my training data are not just short names like "Dog" or "Cat", they are more of lengthy sentences that range from 2 words to around ...
- 103
0
votes
1
answer
20
views
Ways to build Abstractive summarisation and what are it's challenges
What are state of art techniques to build Abstractive summarisation on some paragraphs or articles and what kind of hurdles or challenges are there to approach this problem?
1
vote
1
answer
199
views
How can i extract words from a single concatenated word?
I'm stuck on this problem and would love some input.
I have mulitple words such as getExtention, getPath, someWord or someword and i want to separate each concatinated words into its own words such as:...
0
votes
1
answer
509
views
Using BERT for search engine with an Elastic Database
I want to make Documents search engine where the user will type a query and top n relevant documents should be shown.
I want to use BERT for the searching and the first question is can i use it with ...
- 45
1
vote
1
answer
616
views
why multiple attention heads learn differently
In transformer architecture multi head attention blocks are used. While visualizing their output it can be seen that every layer has learnt different relations of words. e.g., layer 5 has learnt that &...
- 894
3
votes
2
answers
1k
views
Named Entity Recognition with BIO Tagging
I'm trying to implement NER using BIO annotation. For example
"I went to the United States"
[O, O, O, B, I, I]
where B and I denote the beginning and '...
- 31
0
votes
0
answers
304
views
Combine BertForSequenceClassificaion with Additional Features
I'm using BertForSequenceClassification + Pytorch Lightning-Flash for a text classification task.
I want to add additional features besides the text (e.g. categorical features). From what I understand,...
- 23
0
votes
0
answers
35
views
An algorithm to extract the purpose of a document
I want to build an algorithm to extract the purpose of the document (scientific papers for example) by extracting the sentences that state the purpose. I don't have many annotated data so I might use ...
- 35
0
votes
2
answers
280
views
BERT Self-Attention layer
I am trying to use the first individual BertSelfAttention layer for the BERT-base model, but the model I am loading from torch.hub seems to be different then the ...
- 163
2
votes
0
answers
142
views
Imbalance classes in Named Entity Recognition
I am currently working on a NER problem which attempts to extract 2 entities - place-of-interest(POI) and street from an address string in the Indonesian language.
I used IndoBert (available here) and ...
- 121
3
votes
1
answer
6k
views
BERT embedding layer
I am trying to figure how the embedding layer works for the pretrained BERT-base model. I am using pytorch and trying to dissect the following model:
...
- 163
1
vote
1
answer
324
views
Is positional encoding (in transformers) an estimation of the relative positions of words in the training corpus texts?
Is this some kind of estimation of the relative positions of words in the training texts? are they creating some kind of statistical "distribution" of words? is "cat" usually 2 or ...
- 13
2
votes
1
answer
2k
views
Combining textual and numeric features into pre-trained Transformer BERT
I have a dataset with 3 columns:
Text
Meta-data (intending to extract features from it, then use those i.e., numerical features)
Target label
Question 1: How can I use a pre-trained BERT instance on ...
2
votes
0
answers
45
views
Social media text analysis
I'm currently analyzing Korean social media text. The below are the steps of the analysis.
Collect/crawling text data from social media (e.g. Twitter, Facebook), which are related to specific topics.
...
- 33
1
vote
1
answer
979
views
Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)
I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:
...
1
vote
0
answers
435
views
BERT MLM overfitting [closed]
We are training the BERT model on masked language modeling task for the Russian Language. Our dataset consists of 60 mln texts with (128 tokens for each text) from online social networks, ...
- 111
2
votes
3
answers
942
views
Word2vec outperforming BERT, possible?
I'm trying to solve a multilabel classification (dataset is tweet text) using a combination of BERT and CNN. As a benchmark, I'd compare it to other word embeddings, one of which is Word2vec. After ...
- 21
1
vote
1
answer
268
views
BERT is running out of memory in forward pass for my dictionary
Running code from this answer, my BERT is running out for my 4k words dictionary.
I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly:
...
- 137
0
votes
1
answer
37
views
What is the common practice for NLP or text mining for non-English?
A lot of natural language processing tools are pre-trained with corpus in English. What if ones need to analyze, say, Dutch text? The blogs I find online are mostly saying traslating text into English ...
- 145