Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
1 vote
1 answer
318 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
0 votes
0 answers
19 views

NLP - support comments analysis

I am new to NLP and looking for some direction since after all my reading I haven't found a definite approach and the subject matter is vast. The project is to focus on specific fields of support ...
0 votes
0 answers
13 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
0 votes
1 answer
1k views

Is it possible feed BERT to seq2seq encoder/decoder NMT (for low resource language)?

I'm working on NMT model which the input and the target sentences are from the same language (but the grammar differs). I'm planning to pre-train and use BERT since I'm working on small dataset and ...
1 vote
2 answers
41 views

How to get sentiment score for a word in a given dataset

I have a sentiment analysis dataset that is labeled in three categories: positive, negative, and neutral. I also have a list of words (mostly nouns), for which I want to calculate the sentiment value, ...
0 votes
1 answer
175 views

Loading a Model with weights and optimizers without creating an instance in PyTorch

I recently downloaded Camembert Model to fine-tune it for my purpose. Upon unzipping the file the contents are: Upon loading the model.pt file using pytorch: ...
1 vote
1 answer
602 views

HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00

I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead ...
0 votes
0 answers
22 views

How to optimize hyperparameters in Bert?

I am using the BERT model in order to classify stereotypes in sentences. I wanted to know if there is a way to automate the optimization of hyperparameters such as 'epochs', 'batchs' or 'learning rate'...
28 votes
7 answers
48k views

How to get sentence embedding using BERT?

How to get sentence embedding using BERT? ...
1 vote
1 answer
35 views

BERT base uncased required gpu ram

I'm working on an NLP task, using BERT, and I have a little doubt about GPU memory. I already made a model (using DistilBERT) since I had out-of-memory problems with tensorflow on a RTX3090 (24gb gpu'...
0 votes
0 answers
10 views

BertTokenizer on custom data returns same index for all tokens

I'm trying to train Bert tokenizer on a custom dataset but when running tokenizer.tokenize on sample data, it returns the same index for every tokens which is ...
3 votes
1 answer
873 views

How Transformer is Bidirectional - Machine Learning

Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/...
0 votes
1 answer
16 views

How do i generate text from ids in Torchtext's sentencepiece_numericalizer?

The torchtext sentencepiece_numericalizer() outputs a generator with indices SentencePiece model corresponding to token in the input sentence. From the generator, I ...
2 votes
1 answer
255 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
2 votes
3 answers
2k views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
0 votes
0 answers
21 views

Can I use MLM method to fine tune my BERT model, if it was initially trained with natural language inference method?

I am using BERT model for sentence similarity task. However my dataset with sentence is very specific and I want to fine tune my model on it first. My dataset is unlabelled. And BERT model that I want ...
0 votes
0 answers
6 views

Unable to debug where torch Adam optimiser is going wrong

I was implementing a training loop in vscode. I have created a Adam optimizer using XLM-Roberta model as follows: ...
0 votes
0 answers
24 views

Should I pretrain my BERT model on specific dataset if it has only one class of labels?

I want to use BERT model for sentences similarity measuring task. I know that BERT models were trained with natural language inference architecture with dataset with labels neutral, entailment, ...
0 votes
0 answers
23 views

While training BERT variant, getting IndexError: index out of range in self

While training XLMRobertaForSequenceClassification: ...
1 vote
2 answers
635 views

BERT classifier with Ktrain API is unable to predict new data

I have trained a classifier for sentiment analysis using BERT architecture. I am able to train the classifier and I am getting a validation accuracy of 87%. But whenever I feed in test data, or some ...
1 vote
0 answers
20 views

Why shouldn't we mask [CLS] and [SEP] in preparing inputs for a MLM?

I know that MLM is trained for predicting the index of MASK token in the vocabulary list, and I also know that [CLS] stands for the beginning of the sentence and [SEP] telling the model the end of the ...
0 votes
0 answers
13 views

How can I set vocab_size of BertModel(config=configuration).from_pretrained('bert-base-cased') to a higher value?

I have the following issue with the BERT transformer in python: When I feed to BertModel().from_pretrained('bert-base-cased') an input obtained from BertTokenizer.from_pretrained('bert-base-cased') ...
0 votes
1 answer
95 views

how to improve my imbalanced data NLP model?

I want to classify a patient's health as a prediction probability and get the top 10 most ill patients in a hospital. I have patient's condition notes, medical notes, diagnoses notes, and lab notes ...
0 votes
1 answer
15 views

When would you use word2vec over BERT?

I am very new to Machine Learning and I have recently been exposed to word2vec and BERT. From what I know, word2vec provides a vector representation of words, but is limited to its dictionary ...
1 vote
0 answers
13 views

How does bert produce variable output shape?

Suppose if I provide a list of sentences: ['I like python', 'I am learning python', # longest sentence of length 4 tokens 'Python is simple'] Bert will produce ...
0 votes
0 answers
7 views

Question about computing language modeling loss with multi gpu

When training BERT or GPT or other language model, we use the mean of cross entropy as loss function(don't consider label smoothing). Here B denote for batch size, len denote target length of i-th ...
1 vote
0 answers
8 views

Can I use Bert on data subsets and get a compatible representation for the whole dataset?

I need to build an embedding for a massive amount of phrases. I want to use BERT (through the library https://www.sbert.net/). Can I build a partial representation of the data, say encoding 1000 ...
1 vote
1 answer
28 views

Comparison between applications of vanilla transformer and BERT

I try to identify applications of vanilla transformer in nlp, as well as those in BERT. But I don't seem to find good summaries for either of them. Thus my questions are: what are the applications of ...
3 votes
2 answers
366 views

Detecting grammatical errors with BERT

We fine-tuned BERT (bert-base-uncased) model with CoLA dataset for sentence classification task. The dataset is a mix of ...
0 votes
0 answers
42 views

Hugging face Model Output 'last_hidden_state'

I am using the Huggingface BERTModel, The model gives Seq2SeqModelOutput as output. The output contains the past hidden states and the last hidden state. These are my questions What is the use of the ...
0 votes
0 answers
25 views

Special tokens for encoder and decoder in the transformer architecture

I am trying to wrap my head around the different special tokens that the different transformer architectures use. For example, let's say we have the following input and target both for a text ...
0 votes
1 answer
697 views

Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?

I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it ...
0 votes
1 answer
17 views

Which is the difference between the two Greek BERT models?

I want to use the Greek BERT which can be found here https://huggingface.co/nlpaueb/bert-base-greek-uncased-v1 However I am confused about which model should I use and which are the differences. The ...
1 vote
1 answer
1k views

Overfitting in Huggingface's TFBertForSequenceClassification

I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
0 votes
1 answer
190 views

Is it possible to fine-tuning BERT by training it on multiple datasets? (Each dataset having it's own purpose)

BERT can be fine-tuned on a dataset for a specific task. Is it possible to fine-tune it on all these datasets for different tasks and then be utilized for these tasks instead of fine-tuning a BERT ...
1 vote
1 answer
28 views

Is it okay to fine-tuning bert with large context for sequence classification?

I want to create sequence classification bert model. The input of model will be 2 sentence. But i want to fine tuning the model with large context data which consists of multiple sentences(which ...
2 votes
1 answer
611 views

How to classify neutral sentiments using BERT

We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also? https://colab.research.google.com/github/google-...
0 votes
0 answers
11 views

Weighting Sentence Similarity by salience or frequency

It seems like the new standard in text search is sentence or document similarity, using things like BERT sentence embeddings. However, these don't really have a way to consider the salience of ...
1 vote
0 answers
45 views

Can pre-trained transformers (I.e., BERT) handle numerical/spatial data

I’m curious to know if pre-trained transformers could handle search queries that include numerical data or make references to spatial relationships. Take an example dataset of a list of restaurants, ...
0 votes
2 answers
292 views

Smallest Possible Dataset for Text Classification using BERT

What are your experiences for appropriate dataset sizes for usual text classification tasks using a finetuned BERT such as sentiment analysis? ~100 examples ~1000 examples ... ~10000000 examples ...
0 votes
0 answers
111 views

How to improve accuracy? BERT

Dataframe: ...
1 vote
1 answer
460 views

Bert and SVM classification

I'm trying to understand the concepts in the title and how they fit into the task of binary classification. According to my understanding so far, you can encode text using various feature-extraction ...
5 votes
1 answer
225 views

Why do BERT classification do worse with longer sequence length?

I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts ...
0 votes
0 answers
12 views

Using different layers in model output to achieve cosine distance in embedding space

I'm looking at the article of Sentence-BERT, I'm trying to do some embeddings with the same siamese architecture. Later, I will want to compare embeddings from my model with FastText model using ...
0 votes
1 answer
43 views

How to improve the evaluation score for highly imbalanced dataset?

I have trained my BERT model(bert-base-cased) to detect toxic comments. I used the Toxic Comment Classification Challenge dataset from the Kaggle. My accuracy is 98% and the AUROC for various sub-...
0 votes
1 answer
97 views

Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?

I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or ...
2 votes
1 answer
85 views

BERT - The purpose of summing token embedding, positional embedding and segment embedding

I read the implementation of BERT inputs processing (image below). My question is why the author chose to sum up three types of embedding (token embedding, positional embedding and segment embedding)?
0 votes
0 answers
22 views

how can i average subword embedding?

how can i average subword embedding vectors to generate an approximate vector for the original word as i get the embedding using this function ...
0 votes
0 answers
44 views

AraBERT Overfitting for sentiment analysis

I Am newbie to Machine Learning in general. I am currently trying to follow a tutorial on sentiment analysis using BERT and Transformers. I do not know how i can Read the results to know the ...
1 vote
1 answer
128 views

BERT is running out of memory in forward pass for my dictionary

Running code from this answer, my BERT is running out for my 4k words dictionary. I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly: ...

1
2 3 4 5 6