Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

114 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7 votes
1 answer
9k views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
Harman's user avatar
  • 706
3 votes
0 answers
38 views

Weird behaviour when using RobERTA for text classification

I have a dataset with around 70 classes and the dataset is largely balanced ~150 samples per class. I am finetuning RoBERTA-base for 4 epochs with a ...
user1274878's user avatar
3 votes
0 answers
1k views

Combining heterogeneous numerical and text features

We want to solve a regression problem of the form "given two objects $x$ and $y$, predict their score (think about it as a similarity) $w(x,y)$". We have 2 types of features: For each ...
Dmitry's user avatar
  • 81
3 votes
0 answers
188 views

Medical NER for French language

I'm currently exploring the options to extract medical NER specifically for French language. I tried SpaCy's general French NER but it wasn't helpful to the cause (...
Van Peer's user avatar
  • 285
3 votes
2 answers
1k views

Detecting grammatical errors with BERT

We fine-tuned BERT (bert-base-uncased) model with CoLA dataset for sentence classification task. The dataset is a mix of ...
Van Peer's user avatar
  • 285
3 votes
0 answers
60 views

Structuring a LSTM Layer

I'm trying to improve an NER Bert sequence tagger using LSTM layers in TensorFlow. I'm a bit unclear on the interface and how a LSTM layer should be set up. Currently, I'm taking in 3-5 sentences and ...
bbbbbb's user avatar
  • 41
2 votes
0 answers
25 views

Use text embeddings to map job descriptions to ESCO occupations

I'm trying to build a model to map job descriptions to ESCO occupations which is a taxonomy for job titles. Every ESCO occupations have a title, a description and some essential skills. Ideally I ...
GanaelD's user avatar
  • 21
2 votes
0 answers
2k views

How many samples in dataset are required to fine-tune BERT for binary classification?

I'm trying to fine-tune a BERT-based model for a binary classification task (data is in English). The dataset I'm working with is quite small (~500 samples, out of which 80% are currently used for ...
Occasus's user avatar
  • 21
2 votes
1 answer
472 views

Bertopic with embedding: unable to use find_topic

I've used BERTopic with success for the following tasks: get topics, visualise (topics, barcharts, documents ...) and DTM (extended to get area plot with considerable success). However, I am unable to ...
semmyk-research's user avatar
2 votes
0 answers
38 views

Comparing the cosine similarities of the same word representations, from two separate models (vector spaces)

I am comparing the cosine similarities of word representations derived from a BERT model and also from a static Word2Vec model. I understand that the vector spaces of the two models are inherently ...
the_herpe's user avatar
2 votes
0 answers
429 views

How to add custom embeddings for bert

Pretrained Bert input has token embeddings, segment embeddings, position embeddings. But I would like to add some custom embeddings along with them and feed them to pretrained bert. How can I ...
SS Varshini's user avatar
2 votes
0 answers
173 views

Ways to cluster word senses with word embeddings

I'm trying to semantically cluster polysemous words or word with different meanings in a corpus for my class study and I want to do it by word embeddings but I have no Idea how to reach to the ...
amkyp's user avatar
  • 21
2 votes
1 answer
3k views

HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00

I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead ...
JasonExcel's user avatar
2 votes
0 answers
162 views

Imbalance classes in Named Entity Recognition

I am currently working on a NER problem which attempts to extract 2 entities - place-of-interest(POI) and street from an address string in the Indonesian language. I used IndoBert (available here) and ...
tangolin's user avatar
  • 121
2 votes
0 answers
47 views

Social media text analysis

I'm currently analyzing Korean social media text. The below are the steps of the analysis. Collect/crawling text data from social media (e.g. Twitter, Facebook), which are related to specific topics. ...
Inhyeok Yoo's user avatar
2 votes
0 answers
209 views

Loss first decreases and then increases

I am using pre-trained xlnet-base-cased model and training it further on real vs fake news detection dataset. I noticed a trend in accuracy for first epoch. ...
Kanish Anand's user avatar
2 votes
0 answers
272 views

Remove subwords from BERT output

I'm trying to build a multilingual WSD system with BERT on top as the embedding layer. In order to have better performances, after BERT finishes its job (and performs Transfer Learning), I need to ...
Gianmarco F.'s user avatar
2 votes
0 answers
525 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
boredaf's user avatar
  • 161
2 votes
0 answers
702 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
daanvdn's user avatar
  • 21
2 votes
0 answers
221 views

Can BERT embeddings be used to reproduce the original content of the text?

From what I understand, BERT provides contextualized embeddings that are not deterministic the way Word2Vec embeddings (i.e. the word "Queen" doesn't always produce the same vector, it'll be different ...
RafiyaJaved's user avatar
2 votes
1 answer
392 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
Oren Matar's user avatar
2 votes
1 answer
1k views

How to classify neutral sentiments using BERT

We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also? https://colab.research.google.com/github/google-...
Prashanth's user avatar
2 votes
0 answers
223 views

Paragraph Generator using BERT or GPT

I am trying to generate similar sentences, called paragraph generation. For example, what is the name of the eldest brother of ram? - For these paragraphs can be - who is the oldest brother of ram? , ...
Sandeep Bhutani's user avatar
1 vote
0 answers
33 views

Interpretation of Evaluation Values of Augmented SBERT Training with EmbeddingSimilarityEvaluator()

I train a BI-Encoder to get an Augmented SBERT and I get a final training result. How can I interpret the following output of the final training result? ...
Christian01's user avatar
1 vote
0 answers
54 views

I can't get good performance from BERT

I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin I compare an LSTM model with a glove model and a BERT model. I ...
Seydou GORO's user avatar
1 vote
0 answers
138 views

Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding

I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
soph's user avatar
  • 113
1 vote
2 answers
1k views

Fine-tuned MLM based RoBERTa not improving performance

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM. We took some sample data points (2M+) & fine-tuned ...
Kalsi's user avatar
  • 11
1 vote
2 answers
84 views

Fine-tune GPT on sketch data (stroke-3)

These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it. I was planning on using some pretrained models that are ...
ilved17's user avatar
  • 41
1 vote
1 answer
333 views

Is there any concern for a pretrained model to overfitting to a fine-tuning task that has overlapping pretraining and training data?

Let's say my language model is pretrained on a general text corpus, and I want to use it for some specific downstream task that has it's datasets also included in the general corpus, is there any ...
Brian's user avatar
  • 11
1 vote
1 answer
765 views

How does BERT work for Aspect-Based sentiment analysis?

I have recently used a package to perform Aspect-Based Sentiment Analysis (ABSA) through a BERT model. Briefly, the model takes two inputs: words that constitute the aspects a sentence on which we ...
Alberto De Benedittis's user avatar
1 vote
1 answer
402 views

BERTopic Visualization

I new to topic modeling and I'm trying to use BERTopic inside of PyCharm. I'm struggling to ...
Life is complex's user avatar
1 vote
0 answers
482 views

What are the inputs of encoder and decoder layers of transformer architecture?

In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
canP's user avatar
  • 101
1 vote
0 answers
374 views

How does bert produce variable output shape?

Suppose if I provide a list of sentences: ['I like python', 'I am learning python', # longest sentence of length 4 tokens 'Python is simple'] Bert will produce ...
ShaoMin Liu's user avatar
1 vote
0 answers
15 views

Can I use Bert on data subsets and get a compatible representation for the whole dataset?

I need to build an embedding for a massive amount of phrases. I want to use BERT (through the library https://www.sbert.net/). Can I build a partial representation of the data, say encoding 1000 ...
Ben Friedlander's user avatar
1 vote
0 answers
112 views

Can pre-trained transformers (I.e., BERT) handle numerical/spatial data

I’m curious to know if pre-trained transformers could handle search queries that include numerical data or make references to spatial relationships. Take an example dataset of a list of restaurants, ...
Ellio's user avatar
  • 93
1 vote
0 answers
195 views

BERT each Word Embedding in Keras

How to use BERT to extract the embeddings of every word in a sentence. Suppose I pass my corpus of sentences with different lengths to a BERT model , I want to be able to extract the embeddings of ...
Sakher's user avatar
  • 118
1 vote
0 answers
103 views

Getting the keywords of text classification prediction in real time

I am using a BERT based text classification model to classify sentences/documents. My use case is like I need to highlight the words that are responsible for classification of a particular class. This ...
sampath kumaran's user avatar
1 vote
0 answers
36 views

Where to start with ChatBots?

I want to start my journey into ChatBots and how I can create them. I have read some articles regarding the type of chatbots. Basically there are 2 types of chatbots, one is a rule based and the other ...
spectre's user avatar
  • 2,105
1 vote
0 answers
371 views

How to Fine Tune a BERT model for sentiment analysis to get the best f1 score

I am building a multi-class sentiment analysis BERT model that's optimized to give the best f1 score. More specifically, I train each epoch by optimizing binary cross entropy per class, taking the ...
vgoklani's user avatar
  • 238
1 vote
0 answers
217 views

Using Sentence-Bert with other features in scikit-learn

I have a dataset where one feature is text and 4 more features. Sentence-Bert vectorizer transforms text data into tensors. I can use these sparse matrices directly with a machine learning classifier. ...
Narges Se's user avatar
1 vote
0 answers
317 views

NER prections with distilbert transformer model

I am trying to extract 'agreement date' label from a corpus of legal contracts. In the train dataset, I used pytorch-transformer model to train. ...
Jay's user avatar
  • 56
1 vote
1 answer
804 views

Is it possible to fine-tuning BERT by training it on multiple datasets? (Each dataset having it's own purpose)

BERT can be fine-tuned on a dataset for a specific task. Is it possible to fine-tune it on all these datasets for different tasks and then be utilized for these tasks instead of fine-tuning a BERT ...
Tony Jesuthasan's user avatar
1 vote
0 answers
366 views

Is it possible to fine-tune a (Spanish RoBERTa) model for a different task?

I'm doing sentiment analysis of Spanish tweets. After reviewing some of the recent literature, I've seen that there's been a most recent effort to train a RoBERTa model exclusively on Spanish text. It ...
LeLuc's user avatar
  • 131
1 vote
0 answers
43 views

How to do Bert Finetuning of failure cases?

I have a large dataset(public available) of text that is labelled. However the test distribution (actual production setting of company) while similar is not from the same source and thus tends to fail ...
Gary Ong's user avatar
1 vote
0 answers
45 views

NLP-Problem, language model BERT?

Right now I am in the process of deciding on my masters thesis topic. Right now I and my professor are thinking about the possibility of having a large dataset of product requirements given in a ...
sharaku17's user avatar
1 vote
1 answer
1k views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
Bloodstone Programmer's user avatar
1 vote
1 answer
303 views

BERT is running out of memory in forward pass for my dictionary

Running code from this answer, my BERT is running out for my 4k words dictionary. I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly: ...
taciturno's user avatar
  • 137
1 vote
0 answers
27 views

Extracting layer output from Classification model of SimpleTransformer

I have fine tuned a bert base model for text classification task. Now, I want to extract hidden layer output so as to combine this output with other features to train a random forest model. Problem ...
SK Singh's user avatar
1 vote
1 answer
1k views

Bert and SVM classification

I'm trying to understand the concepts in the title and how they fit into the task of binary classification. According to my understanding so far, you can encode text using various feature-extraction ...
Bula's user avatar
  • 111
1 vote
0 answers
549 views

How do I handle class imbalance for text data when using pretrained models like BERT?

I have a skewed dataset consisting of samples of the form: Category 1 10000 Category 2 2000 Category 3 400 Category 4 300 Category 5 100 The dataset ...
nikhil6041's user avatar