Questions tagged [bert]
BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers
357
questions
1
vote
1
answer
1k
views
BertTokenizer Loading Problem
I loaded this BertTokenizer previously, but now it is showing, I have to make sure I don't have a local directory. In my kaggle kernel, I don't have this local directory.
How to solve it?
...
- 13
1
vote
1
answer
137
views
Can I use Sentence-Bert to embed event triples?
I extracted event triples from sentences using OpenIE. Can I concatenate the components in the event triple to make it a sentence and use Sentence-Bert to embed it?
It seems no one has done this way ...
- 165
0
votes
0
answers
762
views
Using BERT instead of word2vec to extract most similar words to a given word
I am fairly new to BERT, and I am willing to test two approaches to get "the most similar words" to a given word to use in Snorkel labeling functions for weak supervision.
Fist approach was ...
0
votes
0
answers
32
views
NLP - support comments analysis
I am new to NLP and looking for some direction since after all my reading I haven't found a definite approach and the subject matter is vast. The project is to focus on specific fields of support ...
- 1
1
vote
1
answer
1k
views
BERT base uncased required gpu ram
I'm working on an NLP task, using BERT, and I have a little doubt about GPU memory.
I already made a model (using DistilBERT) since I had out-of-memory problems with tensorflow on a RTX3090 (24gb gpu'...
- 68
0
votes
1
answer
204
views
How do i generate text from ids in Torchtext's sentencepiece_numericalizer?
The torchtext sentencepiece_numericalizer() outputs a generator with indices SentencePiece model corresponding to token in the input sentence. From the generator, I ...
- 17
0
votes
0
answers
94
views
Should I pretrain my BERT model on specific dataset if it has only one class of labels?
I want to use BERT model for sentences similarity measuring task. I know that BERT models were trained with natural language inference architecture with dataset with labels neutral, entailment, ...
- 183
2
votes
1
answer
562
views
Why shouldn't we mask [CLS] and [SEP] in preparing inputs for a MLM?
I know that MLM is trained for predicting the index of MASK token in the vocabulary list, and I also know that [CLS] stands for the beginning of the sentence and [SEP] telling the model the end of the ...
- 21
0
votes
1
answer
138
views
When would you use word2vec over BERT?
I am very new to Machine Learning and I have recently been exposed to word2vec and BERT.
From what I know, word2vec provides a vector representation of words, but is limited to its dictionary ...
1
vote
0
answers
273
views
How does bert produce variable output shape?
Suppose if I provide a list of sentences:
['I like python',
'I am learning python', # longest sentence of length 4 tokens
'Python is simple']
Bert will produce ...
- 11
1
vote
0
answers
13
views
Can I use Bert on data subsets and get a compatible representation for the whole dataset?
I need to build an embedding for a massive amount of phrases. I want to use BERT (through the library https://www.sbert.net/).
Can I build a partial representation of the data, say encoding 1000 ...
1
vote
1
answer
194
views
Comparison between applications of vanilla transformer and BERT
I try to identify applications of vanilla transformer in nlp, as well as those in BERT. But I don't seem to find good summaries for either of them. Thus my questions are:
what are the applications of ...
- 411
0
votes
0
answers
1k
views
Hugging face Model Output 'last_hidden_state'
I am using the Huggingface BERTModel, The model gives Seq2SeqModelOutput as output. The output contains the past hidden states and the last hidden state.
These are my questions
What is the use of the ...
- 17
0
votes
1
answer
184
views
Which is the difference between the two Greek BERT models?
I want to use the Greek BERT which can be found here https://huggingface.co/nlpaueb/bert-base-greek-uncased-v1
However I am confused about which model should I use and which are the differences.
The ...
- 15
1
vote
1
answer
636
views
Is it okay to fine-tuning bert with large context for sequence classification?
I want to create sequence classification bert model. The input of model will be 2 sentence. But i want to fine tuning the model with large context data which consists of multiple sentences(which ...
- 11
1
vote
0
answers
102
views
Can pre-trained transformers (I.e., BERT) handle numerical/spatial data
I’m curious to know if pre-trained transformers could handle search queries that include numerical data or make references to spatial relationships.
Take an example dataset of a list of restaurants, ...
- 93
3
votes
1
answer
2k
views
BERT - The purpose of summing token embedding, positional embedding and segment embedding
I read the implementation of BERT inputs processing (image below). My question is why the author chose to sum up three types of embedding (token embedding, positional embedding and segment embedding)?
- 133
0
votes
0
answers
140
views
AraBERT Overfitting for sentiment analysis
I Am newbie to Machine Learning in general. I am currently trying to follow a tutorial on sentiment analysis using BERT and Transformers.
I do not know how i can Read the results to know the ...
- 1
0
votes
1
answer
1k
views
Using KerasClassifier for training neural network
I created a simple neural network for binary spam/ham text classification using pretrained BERT transformer. The current pure-keras implementation works fine. I wanted however to plot certain metrics ...
- 289
2
votes
1
answer
1k
views
how to use bert-tiny using transformers?
how can I use BERT-tiny .. I tried to load bert-base-uncased by this line
...
- 25
1
vote
1
answer
926
views
How to prepare texts to BERT/RoBERTa models?
I have an artificial corpus I've built (not a real language) where each document is composed of multiple sentences which again aren't really natural language sentences.
I want to train a language ...
- 141
1
vote
0
answers
193
views
BERT each Word Embedding in Keras
How to use BERT to extract the embeddings of every word in a sentence.
Suppose I pass my corpus of sentences with different lengths to a BERT model , I want to be able to extract the embeddings of ...
- 118
0
votes
1
answer
815
views
Pretrained German BERT
I'm looking for a (well) pretrained BERT Model in German to be adapted in a Keras/TF framework. Ideally with a minimal example on how to fine-tune the model on specific tasks, i.e. text classification!...
- 7,366
0
votes
1
answer
1k
views
What are the differences between bert embedding and flair embedding
I read about BERT embedding model and FLAIR embedding model, and I'm not sure I can tell what are the differences between them ?
...
- 363
1
vote
0
answers
52
views
Which will be best deep learning model for topic classification using NLP [closed]
I have a dataset consisting of two columns [Text, topic_labels].
Topic_labels are of 6 categories for ex: [plants,animals,birds,insects etc]
I would like to build deep learning-based models in order ...
- 11
1
vote
0
answers
90
views
Getting the keywords of text classification prediction in real time
I am using a BERT based text classification model to classify sentences/documents. My use case is like I need to highlight the words that are responsible for classification of a particular class. This ...
-1
votes
2
answers
167
views
What is distillation in Neural Network?
Distillation seems to be a general technique to reduce the size of NLP/NN models. Can anyone help me to understand intuition and how does it work?
- 1,904
1
vote
0
answers
35
views
Where to start with ChatBots?
I want to start my journey into ChatBots and how I can create them. I have read some articles regarding the type of chatbots. Basically there are 2 types of chatbots, one is a rule based and the other ...
- 1,886
0
votes
2
answers
1k
views
Get sentence embeddings of transformer-based models
I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...).
I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-...
- 145
1
vote
1
answer
101
views
does ValueError: 'rat' is not in list means not exist in tokenizer
Does this error means that the word doesn't exist in the tokenizer
return sent.split(" ").index(word)
ValueError: 'rat' is not in list
the code sequences ...
- 17
1
vote
1
answer
2k
views
How can i get the vector of word using BERT?
I need to get word-vectors using BERT and got this function that i think it should be the one i need
...
- 113
0
votes
1
answer
2k
views
Difference between Doc2Vec and BERT
I am trying to understand the difference between Doc2Vec and BERT. I do understand that doc2vec uses a paragraph ID which also serves as a paragraph vector. I am not sure though if that paragraph ID ...
- 23
0
votes
2
answers
652
views
How to categorise customer complaint using NLP
I have a dataset of community complaints and I would like to build a NLP model on those descriptions and tag a category (can be referred for an inspection or Not ie "Not referred) to each of them....
- 103
2
votes
0
answers
35
views
Comparing the cosine similarities of the same word representations, from two separate models (vector spaces)
I am comparing the cosine similarities of word representations derived from a BERT model and also from a static Word2Vec model.
I understand that the vector spaces of the two models are inherently ...
- 21
0
votes
0
answers
29
views
One class classifier for fraud call detection ( in Hindi language) using BERT
I have created a dataset of text files that are nothing but transcripts of fraud call recordings. I want to implement one class classifier as I have only fraud call audio and transcripts but not a ...
2
votes
1
answer
412
views
How to feed a Knowledge Base into Language Models?
I’m a CS undergrad trying to make my way into NLP Research. For some time, I have been wanting to incorporate "everyday commonsense reasoning" within the existing state-of-the-art Language ...
2
votes
2
answers
174
views
How to improve language model ex: BERT on unseen text in training?
I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure.
However, when I remove a ...
- 123
1
vote
1
answer
130
views
How to precompute one sequence in a sequence-pair task when using BERT?
BERT uses separator tokens ([SEP]) to input two sequences for a sequence-pair task. If I understand the BERT architecture correctly, attention is applied to all inputs thus coupling the two sequences ...
1
vote
1
answer
53
views
Twitter Sentiment Analysis: problem in predicting [closed]
I am going to do Sentiment Analysis over some tweets. The goal is to find out which post is with and which one is against a specific topic(which tweet is saying this product is good and which on is ...
- 263
0
votes
0
answers
269
views
Text classification length
I have a set of text examples I need to learn as class A, and they are of varying lengths, say 10 sentences to 1 sentence long. I have to parse a document to find those strings of text that match one ...
- 115
1
vote
1
answer
129
views
Is binary classification the right choice in this case?
I am somewhat new to text classification and I have some questions if you folks can help:
I have some text I need to be able to classify as belonging to a single class or not (usually 1-10 sentences ...
- 115
2
votes
0
answers
389
views
How to add custom embeddings for bert
Pretrained Bert input has token embeddings, segment embeddings, position embeddings. But I would like to add some custom embeddings along with them and feed them to pretrained bert. How can I ...
- 239
8
votes
1
answer
8k
views
BERT vs GPT architectural, conceptual and implemetational differences
In the BERT paper, I learnt that BERT is encoder-only model, that is it involves only transformer encoder blocks.
In the GPT paper, I learnt that GPT is decoder-only model, that is it involves only ...
- 205
2
votes
2
answers
28
views
How to learn common sense constants? Look in body for detail
If I wanted to learn constants for example week -> 7 days, chicken -> 2 legs, day -> 24, 1km -> 1000 meters hours, and so on, would it be possible to extract this information from a BERT ...
- 21
1
vote
1
answer
8k
views
Optimal batch size and number of epoch for BERT
I use this tutorial https://www.tensorflow.org/text/tutorials/classify_text_with_bert and get different accuracy depend on epoch numbers and batch sizes. What's optimal parameters?
- 125
1
vote
0
answers
353
views
How to Fine Tune a BERT model for sentiment analysis to get the best f1 score
I am building a multi-class sentiment analysis BERT model that's optimized to give the best f1 score.
More specifically, I train each epoch by optimizing binary cross entropy per class, taking the ...
- 238
10
votes
2
answers
3k
views
Why should I understand AI architectures?
Why should I understand what is happening deep down in some AI architecture?
For example LSTM-BERT- Partial Conv... Architectures like this. Why should I understand what is going on while I can find ...
- 117
2
votes
0
answers
161
views
Ways to cluster word senses with word embeddings
I'm trying to semantically cluster polysemous words or word with different meanings in a corpus for my class study and I want to do it by word embeddings but I have no Idea how to reach to the ...
- 21
2
votes
0
answers
207
views
Error when trying to predict BERT model and obtaining classification report [closed]
I am following this tutorial about multi-label, multi-class classification using BERT. I am trying to get the predicted classification for y_pred, but when running ...
- 21
0
votes
0
answers
31
views
How do I work with noisy real world text data for text classification?
I have a topic classification model built upon Bert, when I deploy my model people input strings of a random nature like :
"aaaaaa" "aaa bbb" "ab ab ab"
and so on.
My ...
