Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
6 votes
1 answer
10k views

What GPU size do I need to fine tune BERT base cased?

I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
Darshan Bhandari's user avatar
1 vote
1 answer
868 views

Can we use sentence transformers to embed sentences without labels?

I was trying to use this project : https://github.com/UKPLab/sentence-transformers for embedding non english sentences, the language is not a human speaking language, its machine language (x86) but ...
OneAndOnly's user avatar
1 vote
2 answers
2k views

Splitting into multiple heads -- multihead self attention

So, I have a doubt in Attention is all you need: The implementation of transformers on tensorflow's official documentation says: Each multi-head attention block gets three inputs; Q (query), K (key), ...
AVN's user avatar
  • 11
8 votes
2 answers
7k views

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library....
fractalnature's user avatar
2 votes
4 answers
487 views

NLP SBert (Bert) for answer comparison STS

I've been researching a good way to automate short answer evaluation. Essentially a teacher gives a test with some questions like: Question: why did columbus sail westward to find asia? Answer: so he ...
b_the_builder's user avatar
0 votes
1 answer
140 views

Using BERT for co-reference resolving, what's the loss function?

I'm working my way around using BERT for co-reference resolving. I'm following this highly-cited paper BERT for Coreference Resolution: Baselines and Analysis (https://arxiv.org/pdf/1908.09091.pdf). I ...
EyeQ Tech's user avatar
  • 113
3 votes
1 answer
99 views

Are there any objections to using the same (unlabelled) data for pre-training of a BERT-Based model and the downstream task?

I'm looking to train an Electra model using unlabelled data in a specific field. Are there any objections to using the same data for unsupervised learning and then using the same data downstream for ...
user103134's user avatar
1 vote
2 answers
226 views

Problem of continuous training - Supervised learning

I am sure this is a most common problem, but would like to know by experts on how to tackle it. Note that, I mostly deal with textual data (NLP problems). When a supervised learning model is created, ...
Sandeep Bhutani's user avatar
0 votes
1 answer
309 views

Loading a Model with weights and optimizers without creating an instance in PyTorch

I recently downloaded Camembert Model to fine-tune it for my purpose. Upon unzipping the file the contents are: Upon loading the model.pt file using pytorch: ...
MetaInformation's user avatar
4 votes
2 answers
8k views

Can we use BERT for only word embedding and then use SVM/RNN to do intent classification?

According to this article, "Systems used for intent classification contain the following two components: Word embedding, and a classifier." This article also evaluated BERT+SVM and Word2Vec+...
metk's user avatar
  • 41
0 votes
1 answer
2k views

For NLP, is GPT-3 better than RoBERTa? [closed]

I am learning deep learning and I want to get into NLP. I have done LSTM, and now I am learning about vectorisation and transformers. Can you please tell me, which algorithm is more effective and ...
Samar Pratap Singh's user avatar
2 votes
3 answers
5k views

Imbalanced Dataset (Transformers): How to Decide on Class Weights?

I'm using SimpleTranformers to train and evaluate a model. Since the dataset I am using is severely imbalanced, it is recommended that I assign weights to each ...
Aventinus's user avatar
  • 213
0 votes
3 answers
71 views

BERT reasoning capabilities

I'm working on a Twitter classification task and while analyzing the errors I found quite a few strange predictions. I'm searching for a tool (preferably open-source) similar to https://...
moz_szt's user avatar
  • 75
-1 votes
1 answer
92 views

Does BERT pretrain only on masked tokens?

I was a bit confused on the details of the Masked Language Model in BERT pretraining. Does the model only predict the masked tokens for the purposes of pretraining or does it predict it for all tokens?...
rsvarma's user avatar
  • 23
8 votes
3 answers
5k views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
Aaditya ura's user avatar
1 vote
1 answer
537 views

What are the merges and vocab files used for in BERT-based models?

The title says it all. I see plenty online about how to initialize RoBERTa with a merges and vocab file, but what is the point of these files? What exactly are they used for?
Nick Koprowicz's user avatar
2 votes
4 answers
3k views

Next sentence prediction in RoBERTa

I'm trying to wrap my head around the way next sentence prediction works in RoBERTa. Based on their paper, in section 4.2, I understand that in the original BERT they used a pair of text segments ...
Shashi's user avatar
  • 21
1 vote
1 answer
3k views

Overfitting in Huggingface's TFBertForSequenceClassification

I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
Shahad Mahmud's user avatar
0 votes
1 answer
29 views

What is syntax V and S standing for nominal subject?

I was reading the recent paper https://www.aclweb.org/anthology/P19-1580.pdf and noticed that in section 5.2, the syntactic relation is studied in terms of the "direction between two tokens". In table ...
exteral's user avatar
  • 105
0 votes
1 answer
49 views

What are the simplest methods for the label noise problem?

If I have enough low quality data from unsupervised methods or rule-based methods. I read from https://github.com/subeeshvasu/Awesome-Learning-with-Label-Noise ,but these methods are a little complex ...
惊天补扣's user avatar
1 vote
2 answers
624 views

TensorFlow1.15, multi-GPU-1-machine, how to set batch_size?

The input function code: ...
惊天补扣's user avatar
3 votes
1 answer
829 views

German Chatbot or conversational AI

I want to build a chatbot mostly BERT(Transformer) based in the German Language. But I do not find any German chatbot data set! So does it make sense to use google translator API to translate the ...
Bot_Start's user avatar
2 votes
1 answer
3k views

How to use fine tuning of BERT when i have unlabelled dataset of text documents?

I have gained a basic understanding of using BERT for various NLP/text mining tasks. When it comes to fine-tuning of BERT, I always see that fine-tuning is performed using some classification tasks. ...
Vivek Mehta's user avatar
1 vote
0 answers
312 views

Using BERT for input embeddings in a seq2seq model

I'm currently trying to implement a paper that describes using BERT to embed inputs into a seq2seq model. "For word vectors, we use the deep contextualized word vectors from ELMo (Peters et al., ...
Matthew's user avatar
  • 11
9 votes
2 answers
4k views

Is BERT a language model?

Is BERT a language model in the sense of a function that gets a sentence and returns a probability? I know its main usage is sentence embedding, but can it also provide this functionality?
Amit Keinan's user avatar
1 vote
1 answer
3k views

Implementation of BERT using Tensorflow vs PyTorch

BERT is an NLP model developed by Google. The original BERT model is built by the TensorFlow team, there is also a version of BERT which is built using PyTorch. What is the main difference between ...
codeczar's user avatar
  • 153
2 votes
1 answer
2k views

What information does output of [SEP] token captures in BERT?

After reading around on the web I came to understand that the output representation of the special token [CLS] captures the representation of a sentence (am I correct?). My primary question is what ...
Aman Krishna's user avatar
1 vote
0 answers
25 views

How to convert subword PPL to word level PPL?

I'm using this formula to covert subword perpexity to word perplexity: PPL_word = exp(log(PPL_subword) * num_subwords / num_words) The question is do I need to ...
omer sahban's user avatar
7 votes
2 answers
5k views

Does BERT use GLoVE?

From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated. What I don't know ...
birdmw's user avatar
  • 173
0 votes
1 answer
347 views

How to identify topic transition in consecutive sentences using Python?

I'm new to data mining. I want to detect topic transition among consecutive sentences. For instance, I have a paragraph (this could be a collection of dozens of sentences, sometimes without ...
E_learner's user avatar
  • 111
1 vote
0 answers
885 views

System Requirement to train BERT model

How much Hardware is required to train it well?(My current PC specs: 8GB RAM, i5 2 core Processor, Standard GPU (No work going on GPU)) I have a dataset of approx 1lakh records.Is it is necessary to ...
Sweety's user avatar
  • 13
1 vote
0 answers
584 views

Training PCA on BERT word embedding: entire training dataset or each document?

I want to reduce the dimensionality of the BERT word embedding to, let's say, 50 dimensions. I am trying with PCA. I will use that for the document classification task. Now for training PCA, should ...
user3363813's user avatar
1 vote
2 answers
840 views

Can BERT be used for predicting words?

I have a question regarding the pre-training section (in particular, the Masked Language Model). In the example Let's stick to improvisation in this skit, by masking the word improvisation, after ...
moz_szt's user avatar
  • 75
1 vote
2 answers
714 views

BERT classifier with Ktrain API is unable to predict new data

I have trained a classifier for sentiment analysis using BERT architecture. I am able to train the classifier and I am getting a validation accuracy of 87%. But whenever I feed in test data, or some ...
Piyush Dongre's user avatar
1 vote
1 answer
218 views

How can I tokenize a text file with BERT or something similar?

I want to use the twitter datasets in a project and the tweet contents look something like this: ...
DataDude123's user avatar
1 vote
2 answers
3k views

Difference between using BERT as a 'feature extractor' and fine tuning BERT with its layers fixed

I understand that there are two ways of leveraging BERT for some NLP classification task: BERT might perform ‘feature extraction’ and its output is input further to another (classification) model ...
MilaHalina 's user avatar
2 votes
0 answers
266 views

Remove subwords from BERT output

I'm trying to build a multilingual WSD system with BERT on top as the embedding layer. In order to have better performances, after BERT finishes its job (and performs Transfer Learning), I need to ...
Gianmarco F.'s user avatar
1 vote
3 answers
3k views

Fastest way for 1 vs all lookup on embeddings

I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array. I wish to compare a new unseen text to all the 1 ...
Isbister's user avatar
  • 183
8 votes
2 answers
3k views

What should be the labels for subword tokens in BERT for NER task?

For any NER task, we need a sequence of words and their corresponding labels. To extract features for these words from BERT, they need to be tokenized into subwords. For example, the word ...
PinkBanter's user avatar
5 votes
3 answers
11k views

Generating synonyms or similar words from multiples word embeddings

I am looking for a way to generate synonyms, using word embeddings. From one word, and from multiple words. Such as the two example below: "word" -> Word embedding -> generate synonym of "word" "...
David N's user avatar
  • 151
3 votes
2 answers
143 views

How to generate a sentence with exactly N words?

Thanks to GPT2 pretrained model now it is possible to generate meaningful sequence of words with or without prefix. However a sentence should end with a proper endings (.,!,?). I am just wondering how ...
user185597's user avatar
3 votes
1 answer
518 views

BERT in production

I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
illuminato's user avatar
2 votes
1 answer
2k views

Can I fine-tune BERT, ELMO or XLnet for Seq2Seq neural machine translation?

I'm working on neural machine translator that translates English sentences to American sign language sentences(e.g below). I've a quite small dataset - around 1000 sentence pairs. I'm wondering if it ...
NLP Dude's user avatar
0 votes
1 answer
2k views

Is it possible feed BERT to seq2seq encoder/decoder NMT (for low resource language)?

I'm working on NMT model which the input and the target sentences are from the same language (but the grammar differs). I'm planning to pre-train and use BERT since I'm working on small dataset and ...
NLP Dude's user avatar
5 votes
1 answer
1k views

BertPunc (punctuation restoration with BERT)

I've found the script for punctuation restoration. And I have one question about this method. I will briefly explain the logic of the author. One of four tokens is assigned for each word: Other (0), ...
illuminato's user avatar
1 vote
0 answers
36 views

Multimodal end-to-end deep learning

I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains ...
shaun's user avatar
  • 111
2 votes
2 answers
3k views

Semantic text similarity using BERT

Given two sentences, I want to quantify the degree of similarity between the two text-based on Semantic similarity. Semantic Textual Similarity (STS) assesses the degree to which two sentences are ...
Devarshi Goswami's user avatar
3 votes
2 answers
3k views

What are the elements in a BERT word embedding?

As far as I understand, BERT is a word embedding that can be fine-tuned or used directly. With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one ...
Emil's user avatar
  • 179
4 votes
1 answer
450 views

BERT word embedings for finding word definition

Can BERT, GPT or other contextualised embedings be used for finding word definitions? What would be the most effective and not complicated approach for tackling a sample task as described below. Map ...
piernik's user avatar
  • 51
3 votes
1 answer
13k views

What is a 'hidden state' in BERT output?

I'm trying to understand the workings and output of BERT, and I'm wondering how/why each layer of BERT has a 'hidden state'. I understand what RNN's have a 'hidden state' that gets passed to each ...
Nick Koprowicz's user avatar

1
4 5
6
7 8