Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
0
votes
0answers
15 views

What is the typical accuracy of masked language models during BERT pretraining?

I was reading the BERT paper but I didn't find any tables concerning the performance of the masked language models during pretraining. Does anyone know the accuracy of BERT's masked language model?
3
votes
1answer
85 views

Bert for QuestionAnswering input exceeds 512

I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long ...
0
votes
0answers
14 views

Bert question answering start_positions is larger than end_positions

I want to fine-tune Bert on question answering for a closed domain, so I started by discovering how it works first, i executed the code bellow but the result is no write, the start position is larger ...
1
vote
0answers
20 views

Does BERT has any advantage over GPT3?

I have read a couple of documents that explain in detail about the greater edge that GPT-3(Generative Pre-trained Transformer-3) has over BERT(Bidirectional Encoder Representation from Transformers). ...
1
vote
0answers
24 views

Question about BERT embeddings with high cosine similarity

Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?
1
vote
1answer
26 views

Does finetuning BERT involving updating all of the parameters or just the final classification layer?

Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP. But during finetuning, for example trying ...
0
votes
2answers
43 views

Getting sentence embeddings with sentence_transformers

I have a text column in my data frame which contains paragraph(s) having multiple and variable sentences in each instance/example/row of the dataframe. Then, I created the sentence tokens of that ...
0
votes
0answers
16 views

How is Large BERT is less accurate than basic BERT?

I'm using BERT for text classification in this NLP competition. When I use Basic BERT with 12 layers, 3 epochs, and 32 batch sizes, I get a training accuracy of about 0.84 and a val_accuracy of about ...
2
votes
0answers
46 views

Loss first decreases and then increases

I am using pre-trained xlnet-base-cased model and training it further on real vs fake news detection dataset. I noticed a trend in accuracy for first epoch. ...
0
votes
0answers
13 views

What GPU size do I need to fine tune BERT base cased?

I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
1
vote
1answer
28 views

Can we use sentence transformers to embed sentences without labels?

I was trying to use this project : https://github.com/UKPLab/sentence-transformers for embedding non english sentences, the language is not a human speaking language, its machine language (x86) but ...
0
votes
1answer
21 views

Splitting into multiple heads — multihead self attention

So, I have a doubt in Attention is all you need: The implementation of transformers on tensorflow's official documentation says: Each multi-head attention block gets three inputs; Q (query), K (key), ...
1
vote
2answers
45 views

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library....
2
votes
3answers
64 views

NLP SBert (Bert) for answer comparison STS

I've been researching a good way to automate short answer evaluation. Essentially a teacher gives a test with some questions like: Question: why did columbus sail westward to find asia? Answer: so he ...
0
votes
0answers
19 views

Are there any deep models that are better than BERT-CRF in NER task?

Named entity recognition (NER) is task that mark tags of the input text sequence. BERT-CRF is a good NER model. I want to find a better NER model. Or I want to improve the BERT-CRF model. What can I ...
0
votes
1answer
21 views

Using BERT for co-reference resolving, what's the loss function?

I'm working my way around using BERT for co-reference resolving. I'm following this highly-cited paper BERT for Coreference Resolution: Baselines and Analysis (https://arxiv.org/pdf/1908.09091.pdf). I ...
1
vote
1answer
18 views

Are there any objections to using the same (unlabelled) data for pre-training of a BERT-Based model and the downstream task?

I'm looking to train an Electra model using unlabelled data in a specific field. Are there any objections to using the same data for unsupervised learning and then using the same data downstream for ...
0
votes
0answers
15 views

How to go about fine-tuning BERT using a next-sentence task

I've got a large corpus of documents, and I want to use bert to generate embeddings for a variety of predictive tasks. The documents are multi-sentence, in a non-standard domain, and have labels at ...
1
vote
2answers
24 views

Problem of continuous training - Supervised learning

I am sure this is a most common problem, but would like to know by experts on how to tackle it. Note that, I mostly deal with textual data (NLP problems). When a supervised learning model is created, ...
0
votes
1answer
14 views

Loading a Model with weights and optimizers without creating an instance in PyTorch

I recently downloaded Camembert Model to fine-tune it for my purpose. Upon unzipping the file the contents are: Upon loading the model.pt file using pytorch: ...
1
vote
2answers
126 views

Can we use BERT for only word embedding and then use SVM/RNN to do intent classification?

According to this article, "Systems used for intent classification contain the following two components: Word embedding, and a classifier." This article also evaluated BERT+SVM and Word2Vec+...
-1
votes
1answer
94 views

For NLP, is GPT-3 better than RoBERTa? [closed]

I am learning deep learning and I want to get into NLP. I have done LSTM, and now I am learning about vectorisation and transformers. Can you please tell me, which algorithm is more effective and ...
1
vote
2answers
59 views

Imbalanced Dataset (Transformers): How to Decide on Class Weights?

I'm using SimpleTranformers to train and evaluate a model. Since the dataset I am using is severely imbalanced, it is recommended that I assign weights to each ...
0
votes
0answers
29 views

huggingface bert - need help on working of code

Looking for some explanation of understanding of the BERT implementation by huggingface. I would explain my understanding below and then ask question: Below is code for question answering ...
0
votes
0answers
11 views

Question Answering - Was there any direct model before BERT?

This is a follow up question after question-answering-without-bert-transformers . While in search of a question answering mechanism without transformers, I am hitting dead ends. Old question answering ...
0
votes
1answer
29 views

BERT reasoning capabilities

I'm working on a Twitter classification task and while analyzing the errors I found quite a few strange predictions. I'm searching for a tool (preferably open-source) similar to https://...
0
votes
0answers
138 views

How to fine-tune BERT for Question Answering?

I wish to train two domain-specific models: Domain 1: Constitution and related Legal Documents Domain 2: Technical and related documents. For Domain 1, I've access to a text-corpus with texts from ...
-1
votes
1answer
18 views

Does BERT pretrain only on masked tokens?

I was a bit confused on the details of the Masked Language Model in BERT pretraining. Does the model only predict the masked tokens for the purposes of pretraining or does it predict it for all tokens?...
0
votes
0answers
36 views
0
votes
1answer
80 views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
1
vote
1answer
23 views

What are the merges and vocab files used for in BERT-based models?

The title says it all. I see plenty online about how to initialize RoBERTa with a merges and vocab file, but what is the point of these files? What exactly are they used for?
1
vote
2answers
101 views

Next sentence prediction in RoBERTa

I'm trying to wrap my head around the way next sentence prediction works in RoBERTa. Based on their paper, in section 4.2, I understand that in the original BERT they used a pair of text segments ...
0
votes
1answer
101 views

Overfitting in Huggingface's TFBertForSequenceClassification

I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
0
votes
1answer
20 views

What is syntax V and S standing for nominal subject?

I was reading the recent paper https://www.aclweb.org/anthology/P19-1580.pdf and noticed that in section 5.2, the syntactic relation is studied in terms of the "direction between two tokens". In table ...
0
votes
1answer
27 views

What is the principle of Unsupervised Data Augmentation (UDA)? Why does UDA work?

UDA(https://github.com/google-research/uda) could achieve good accuracy by only 20 training data on text classification. But I find it is hard to reproduce the result on my own dataset. So I want to ...
0
votes
0answers
18 views

What are the simplest methods for the label noise problem?

If I have enough low quality data from unsupervised methods or rule-based methods. I read from https://github.com/subeeshvasu/Awesome-Learning-with-Label-Noise ,but these methods are a little complex ...
0
votes
1answer
31 views

Data quantity is not low but data quality is low, what are the best practices now?

Text classification task, if data quantity is low but data quality is not low. We could use data augment methods for improvement. But the situation is that data quantity is not low and data quality ...
1
vote
1answer
35 views

TensorFlow1.15, multi-GPU-1-machine, how to set batch_size?

The input function code: ...
0
votes
0answers
13 views

Why does the BERT NSP head linear layer have two outputs?

Here's the code in question. https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491 ...
2
votes
1answer
57 views

German Chatbot or conversational AI

I want to build a chatbot mostly BERT(Transformer) based in the German Language. But I do not find any German chatbot data set! So does it make sense to use google translator API to translate the ...
0
votes
1answer
96 views

How to use fine tuning of BERT when i have unlabelled dataset of text documents?

I have gained a basic understanding of using BERT for various NLP/text mining tasks. When it comes to fine-tuning of BERT, I always see that fine-tuning is performed using some classification tasks. ...
0
votes
0answers
15 views

Calculating accuracy in Extractive Summarization

I am trying to implement a text summarization model. I am using keras tensorflow anad I have used bert sentence embeddings and the output of the embeddings are feeded into an LSTM and then to a Dense ...
1
vote
0answers
55 views

Using BERT for input embeddings in a seq2seq model

I'm currently trying to implement a paper that describes using BERT to embed inputs into a seq2seq model. "For word vectors, we use the deep contextualized word vectors from ELMo (Peters et al., ...
0
votes
0answers
123 views

Custom NER with BERT

I want to train bert for a custom entity, and wanted to confirm the correct input format. The common dataset run is the coNLL, which is formatted like this: ...
3
votes
1answer
274 views

Is BERT a language model?

Is BERT a language model in the sense of a function that gets a sentence and returns a probability? I know its main usage is sentence embedding, but can it also provide this functionality?
1
vote
1answer
248 views

Implementation of BERT using Tensorflow vs PyTorch

BERT is an NLP model developed by Google. The original BERT model is built by Tensorflow team there is also a version of BERT which is built using PyTorch. What is the main difference between these ...
0
votes
1answer
155 views

What information does output of [SEP] token captures in BERT?

After reading around on the web I came to understand that the output representation of the special token [CLS] captures the representation of a sentence (am I correct?). My primary question is what ...
1
vote
0answers
11 views

How to convert subword PPL to word level PPL?

I'm using this formula to covert subword perpexity to word perplexity: PPL_word = exp(log(PPL_subword) * num_subwords / num_words) The question is do I need to ...
1
vote
2answers
139 views

Does BERT use GLoVE?

From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated. What I don't know ...
0
votes
1answer
31 views

How to identify topic transition in consecutive sentences using Python?

I'm new to data mining. I want to detect topic transition among consecutive sentences. For instance, I have a paragraph (this could be a collection of dozens of sentences, sometimes without ...