Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
user avatar
0 votes
0 answers
15 views

How to optimize hyperparameters in Bert?

I am using the BERT model in order to classify stereotypes in sentences. I wanted to know if there is a way to automate the optimization of hyperparameters such as 'epochs', 'batchs' or 'learning rate'...
user avatar
  • 320
1 vote
1 answer
30 views

BERT base uncased required gpu ram

I'm working on an NLP task, using BERT, and I have a little doubt about GPU memory. I already made a model (using DistilBERT) since I had out-of-memory problems with tensorflow on a RTX3090 (24gb gpu'...
user avatar
  • 68
0 votes
0 answers
8 views

BertTokenizer on custom data returns same index for all tokens

I'm trying to train Bert tokenizer on a custom dataset but when running tokenizer.tokenize on sample data, it returns the same index for every tokens which is ...
user avatar
  • 257
0 votes
1 answer
16 views

How do i generate text from ids in Torchtext's sentencepiece_numericalizer?

The torchtext sentencepiece_numericalizer() outputs a generator with indices SentencePiece model corresponding to token in the input sentence. From the generator, I ...
user avatar
  • 17
0 votes
0 answers
20 views

Can I use MLM method to fine tune my BERT model, if it was initially trained with natural language inference method?

I am using BERT model for sentence similarity task. However my dataset with sentence is very specific and I want to fine tune my model on it first. My dataset is unlabelled. And BERT model that I want ...
user avatar
  • 113
0 votes
0 answers
6 views

Unable to debug where torch Adam optimiser is going wrong

I was implementing a training loop in vscode. I have created a Adam optimizer using XLM-Roberta model as follows: ...
user avatar
  • 137
0 votes
0 answers
23 views

Should I pretrain my BERT model on specific dataset if it has only one class of labels?

I want to use BERT model for sentences similarity measuring task. I know that BERT models were trained with natural language inference architecture with dataset with labels neutral, entailment, ...
user avatar
  • 113
0 votes
0 answers
18 views

While training BERT variant, getting IndexError: index out of range in self

While training XLMRobertaForSequenceClassification: ...
user avatar
  • 137
1 vote
0 answers
16 views

Why shouldn't we mask [CLS] and [SEP] in preparing inputs for a MLM?

I know that MLM is trained for predicting the index of MASK token in the vocabulary list, and I also know that [CLS] stands for the beginning of the sentence and [SEP] telling the model the end of the ...
user avatar
  • 11
0 votes
0 answers
12 views

How can I set vocab_size of BertModel(config=configuration).from_pretrained('bert-base-cased') to a higher value?

I have the following issue with the BERT transformer in python: When I feed to BertModel().from_pretrained('bert-base-cased') an input obtained from BertTokenizer.from_pretrained('bert-base-cased') ...
user avatar
0 votes
1 answer
15 views

When would you use word2vec over BERT?

I am very new to Machine Learning and I have recently been exposed to word2vec and BERT. From what I know, word2vec provides a vector representation of words, but is limited to its dictionary ...
user avatar
1 vote
0 answers
13 views

How does bert produce variable output shape?

Suppose if I provide a list of sentences: ['I like python', 'I am learning python', # longest sentence of length 4 tokens 'Python is simple'] Bert will produce ...
user avatar
0 votes
0 answers
7 views

Question about computing language modeling loss with multi gpu

When training BERT or GPT or other language model, we use the mean of cross entropy as loss function(don't consider label smoothing). Here B denote for batch size, len denote target length of i-th ...
user avatar
1 vote
0 answers
8 views

Can I use Bert on data subsets and get a compatible representation for the whole dataset?

I need to build an embedding for a massive amount of phrases. I want to use BERT (through the library https://www.sbert.net/). Can I build a partial representation of the data, say encoding 1000 ...
user avatar
1 vote
1 answer
26 views

Comparison between applications of vanilla transformer and BERT

I try to identify applications of vanilla transformer in nlp, as well as those in BERT. But I don't seem to find good summaries for either of them. Thus my questions are: what are the applications of ...
user avatar
  • 359
0 votes
0 answers
39 views

Hugging face Model Output 'last_hidden_state'

I am using the Huggingface BERTModel, The model gives Seq2SeqModelOutput as output. The output contains the past hidden states and the last hidden state. These are my questions What is the use of the ...
user avatar
  • 17
0 votes
0 answers
22 views

Special tokens for encoder and decoder in the transformer architecture

I am trying to wrap my head around the different special tokens that the different transformer architectures use. For example, let's say we have the following input and target both for a text ...
user avatar
0 votes
1 answer
16 views

Which is the difference between the two Greek BERT models?

I want to use the Greek BERT which can be found here https://huggingface.co/nlpaueb/bert-base-greek-uncased-v1 However I am confused about which model should I use and which are the differences. The ...
user avatar
1 vote
1 answer
26 views

Is it okay to fine-tuning bert with large context for sequence classification?

I want to create sequence classification bert model. The input of model will be 2 sentence. But i want to fine tuning the model with large context data which consists of multiple sentences(which ...
user avatar
  • 11
0 votes
0 answers
9 views

Weighting Sentence Similarity by salience or frequency

It seems like the new standard in text search is sentence or document similarity, using things like BERT sentence embeddings. However, these don't really have a way to consider the salience of ...
user avatar
1 vote
0 answers
42 views

Can pre-trained transformers (I.e., BERT) handle numerical/spatial data

I’m curious to know if pre-trained transformers could handle search queries that include numerical data or make references to spatial relationships. Take an example dataset of a list of restaurants, ...
user avatar
  • 93
0 votes
0 answers
12 views

Using different layers in model output to achieve cosine distance in embedding space

I'm looking at the article of Sentence-BERT, I'm trying to do some embeddings with the same siamese architecture. Later, I will want to compare embeddings from my model with FastText model using ...
user avatar
  • 1
2 votes
1 answer
71 views

BERT - The purpose of summing token embedding, positional embedding and segment embedding

I read the implementation of BERT inputs processing (image below). My question is why the author chose to sum up three types of embedding (token embedding, positional embedding and segment embedding)?
user avatar
  • 123
0 votes
0 answers
20 views

how can i average subword embedding?

how can i average subword embedding vectors to generate an approximate vector for the original word as i get the embedding using this function ...
user avatar
  • 15
0 votes
0 answers
37 views

AraBERT Overfitting for sentiment analysis

I Am newbie to Machine Learning in general. I am currently trying to follow a tutorial on sentiment analysis using BERT and Transformers. I do not know how i can Read the results to know the ...
user avatar
  • 1
0 votes
0 answers
69 views

KerasClassifier with random search: can't pickle _thread.RLock objects

I created a simple neural network for binary spam/ham text classification using pretrained BERT transformer. Now I want to apply randomized search for tuning the hyperparameters. For now the only ...
user avatar
  • 257
0 votes
1 answer
20 views

Using KerasClassifier for training neural network

I created a simple neural network for binary spam/ham text classification using pretrained BERT transformer. The current pure-keras implementation works fine. I wanted however to plot certain metrics ...
user avatar
  • 257
1 vote
1 answer
69 views

how to use bert-tiny using transformers?

how can I use BERT-tiny .. I tried to load bert-base-uncased by this line ...
user avatar
  • 15
1 vote
1 answer
72 views

How to prepare texts to BERT/RoBERTa models?

I have an artificial corpus I've built (not a real language) where each document is composed of multiple sentences which again aren't really natural language sentences. I want to train a language ...
user avatar
  • 141
0 votes
0 answers
6 views

Can Domain-Adaption improve the performance of Sentiment Analysis?

Does Domain Adaption have any effect of results in Sentiment Analysis? I am going to train a BERT language model based on some texts particularly in Health area, then I want to apply Opinion Mining on ...
user avatar
0 votes
0 answers
21 views

Transformer model comparison for binary sentiment classification

On two independent datasets, I am comparing XLNet and BERT models with binary sentiment classification tasks: the Twitter dataset, where sentences are short, and the IMDB review dataset, where ...
user avatar
0 votes
0 answers
74 views

Combine different datasets for ensemble model in Keras

Dataset1 is a list of comments with 30 classes (~130k records) from Source 1. Dataset2 is list of comments with 2 classes (~50k records) from source 2. Since ML model wasn't giving good results just ...
user avatar
  • 1
1 vote
0 answers
38 views

BERT each Word Embedding in Keras

How to use BERT to extract the embeddings of every word in a sentence. Suppose I pass my corpus of sentences with different lengths to a BERT model , I want to be able to extract the embeddings of ...
user avatar
  • 13
0 votes
1 answer
48 views

Pretrained German BERT

I'm looking for a (well) pretrained BERT Model in German to be adapted in a Keras/TF framework. Ideally with a minimal example on how to fine-tune the model on specific tasks, i.e. text classification!...
user avatar
  • 6,872
0 votes
0 answers
16 views

How was the vocab built using WordPiece for the paper exBERT?

In the paper exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources the authors point out that: First, we derive an extension vocabulary from the ...
user avatar
  • 313
0 votes
1 answer
171 views

What are the differences between bert embedding and flair embedding

I read about BERT embedding model and FLAIR embedding model, and I'm not sure I can tell what are the differences between them ? ...
user avatar
1 vote
0 answers
35 views

Which will be best deep learning model for topic classification using NLP [closed]

I have a dataset consisting of two columns [Text, topic_labels]. Topic_labels are of 6 categories for ex: [plants,animals,birds,insects etc] I would like to build deep learning-based models in order ...
user avatar
  • 11
1 vote
0 answers
41 views

Getting the keywords of text classification prediction in real time

I am using a BERT based text classification model to classify sentences/documents. My use case is like I need to highlight the words that are responsible for classification of a particular class. This ...
user avatar
0 votes
0 answers
26 views

mismatch between shapes using BERT

I used PyTorch to get word-embeddings using BERT for my sentences which are 150074 sentences....
user avatar
-1 votes
2 answers
55 views

What is distillation in Neural Network?

Distillation seems to be a general technique to reduce the size of NLP/NN models. Can anyone help me to understand intuition and how does it work?
user avatar
  • 1,603
0 votes
0 answers
25 views

Where to start with ChatBots?

I want to start my journey into ChatBots and how I can create them. I have read some articles regarding the type of chatbots. Basically there are 2 types of chatbots, one is a rule based and the other ...
user avatar
  • 1,241
0 votes
1 answer
23 views

Get sentence embeddings of transformer-based models

I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...). I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-...
user avatar
  • 145
1 vote
1 answer
25 views

does ValueError: 'rat' is not in list means not exist in tokenizer

Does this error means that the word doesn't exist in the tokenizer return sent.split(" ").index(word) ValueError: 'rat' is not in list the code sequences ...
user avatar
0 votes
1 answer
186 views

How can i get the vector of word using BERT?

I need to get word-vectors using BERT and got this function that i think it should be the one i need ...
user avatar
0 votes
1 answer
373 views

Difference between Doc2Vec and BERT

I am trying to understand the difference between Doc2Vec and BERT. I do understand that doc2vec uses a paragraph ID which also serves as a paragraph vector. I am not sure though if that paragraph ID ...
user avatar
  • 23
0 votes
0 answers
12 views

NLP with what to replace names in sentences?

My task is named entity-sentiment analysis, and I see if I change the name in the sentence then sentiment can change. Are there any methods to avoid this problem? I think to replace these words with ...
user avatar
0 votes
2 answers
91 views

How to categorise customer complaint using NLP

I have a dataset of community complaints and I would like to build a NLP model on those descriptions and tag a category (can be referred for an inspection or Not ie "Not referred) to each of them....
user avatar
  • 103
2 votes
0 answers
18 views

Comparing the cosine similarities of the same word representations, from two separate models (vector spaces)

I am comparing the cosine similarities of word representations derived from a BERT model and also from a static Word2Vec model. I understand that the vector spaces of the two models are inherently ...
user avatar
0 votes
0 answers
24 views

One class classifier for fraud call detection ( in Hindi language) using BERT

I have created a dataset of text files that are nothing but transcripts of fraud call recordings. I want to implement one class classifier as I have only fraud call audio and transcripts but not a ...
user avatar

1
2 3 4 5 6