Skip to main content

Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
0 votes
1 answer
248 views

When would you use word2vec over BERT?

I am very new to Machine Learning and I have recently been exposed to word2vec and BERT. From what I know, word2vec provides a vector representation of words, but is limited to its dictionary ...
1 vote
2 answers
1k views

Fine-tuned MLM based RoBERTa not improving performance

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM. We took some sample data points (2M+) & fine-tuned ...
7 votes
1 answer
9k views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
2 votes
1 answer
3k views

HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00

I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead ...
1 vote
1 answer
1k views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
2 votes
1 answer
396 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
0 votes
0 answers
20 views

Fine-tuning pretrained model on 2 tasks with 2 labeled dataset

I am having difficulty using BERT for a sentiment analysis task that handles both aspect-based sentiment analysis (ABSA) and comment sentiment analysis. I know that using two separate classification ...
1 vote
1 answer
808 views

Is it possible to fine-tuning BERT by training it on multiple datasets? (Each dataset having it's own purpose)

BERT can be fine-tuned on a dataset for a specific task. Is it possible to fine-tune it on all these datasets for different tasks and then be utilized for these tasks instead of fine-tuning a BERT ...
1 vote
1 answer
427 views

BERTopic Visualization

I new to topic modeling and I'm trying to use BERTopic inside of PyCharm. I'm struggling to ...
1 vote
2 answers
85 views

Fine-tune GPT on sketch data (stroke-3)

These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it. I was planning on using some pretrained models that are ...
0 votes
2 answers
208 views

How to improve the evaluation score for highly imbalanced dataset?

I have trained my BERT model(bert-base-cased) to detect toxic comments. I used the Toxic Comment Classification Challenge dataset from the Kaggle. My accuracy is 98% and the AUROC for various sub-...
1 vote
1 answer
848 views

How does BERT work for Aspect-Based sentiment analysis?

I have recently used a package to perform Aspect-Based Sentiment Analysis (ABSA) through a BERT model. Briefly, the model takes two inputs: words that constitute the aspects a sentence on which we ...
1 vote
2 answers
718 views

BERT classifier with Ktrain API is unable to predict new data

I have trained a classifier for sentiment analysis using BERT architecture. I am able to train the classifier and I am getting a validation accuracy of 87%. But whenever I feed in test data, or some ...
5 votes
1 answer
12k views

How pre-trained BERT model generates word embeddings for out of vocabulary words?

Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
0 votes
0 answers
12 views

How can self-attention be used to combine representations from long text?

The paper "How to Fine-Tune BERT for Text Classification?" discusses using self-attention to combine the representations of a long input text that has been broken into chunks (section 5.3.1)....
1 vote
1 answer
36 views

Reducing emails token count preprocessing for Large Email Datasets - Feeding LLMs

I have a large email dataset in .txt format and want to feed LLMs (like Gemini and ChatGPT) to provide answers based on email content. The token count for my email data is very high (~1M for 1K emails)...
0 votes
1 answer
813 views

Is there bias in matrix multiplications for self attention

When the query matrix Q is computed as $XW_Q$, ($W_Q$ is the weight matrix for the queries), is it implemented as a linear layer without bias? I see some blogs saying there is are bias terms as well. ...
0 votes
1 answer
1k views

Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?

I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it ...
3 votes
0 answers
43 views

Weird behaviour when using RobERTA for text classification

I have a dataset with around 70 classes and the dataset is largely balanced ~150 samples per class. I am finetuning RoBERTA-base for 4 epochs with a ...
0 votes
1 answer
316 views

how to improve my imbalanced data NLP model?

I want to classify a patient's health as a prediction probability and get the top 10 most ill patients in a hospital. I have patient's condition notes, medical notes, diagnoses notes, and lab notes ...
2 votes
0 answers
27 views

Use text embeddings to map job descriptions to ESCO occupations

I'm trying to build a model to map job descriptions to ESCO occupations which is a taxonomy for job titles. Every ESCO occupations have a title, a description and some essential skills. Ideally I ...
1 vote
1 answer
3k views

Overfitting in Huggingface's TFBertForSequenceClassification

I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
2 votes
1 answer
1k views

How to classify neutral sentiments using BERT

We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also? https://colab.research.google.com/github/google-...
0 votes
0 answers
11 views

Does Fine Tuning with Custom Label Build Upon the Capability of Zero Shot Classification or Does It Train from Scratch?

The task is to classify email text bodies into exclusive categories like feedback, complaint etc. I have a labelled dataset available having about 350 samples. I have tried the ...
3 votes
2 answers
1k views

Detecting grammatical errors with BERT

We fine-tuned BERT (bert-base-uncased) model with CoLA dataset for sentence classification task. The dataset is a mix of ...
2 votes
2 answers
699 views

How to get sentiment score for a word in a given dataset

I have a sentiment analysis dataset that is labeled in three categories: positive, negative, and neutral. I also have a list of words (mostly nouns), for which I want to calculate the sentiment value, ...
7 votes
3 answers
10k views

Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?

It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving ...
0 votes
0 answers
7 views

Subsequence classification

Given multiple paragraphs, is it possible to classify an entire paragraph while taking into account the surrounding paragraphs? Paragraph1 Paragraph2 Paragraph3 ...
1 vote
1 answer
306 views

BERT is running out of memory in forward pass for my dictionary

Running code from this answer, my BERT is running out for my 4k words dictionary. I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly: ...
1 vote
1 answer
2k views

Bert and SVM classification

I'm trying to understand the concepts in the title and how they fit into the task of binary classification. According to my understanding so far, you can encode text using various feature-extraction ...
0 votes
0 answers
9 views

Will hypermeters tuned on sampled dataset work for the whole dataset?

I'm doing multi-label classification on text data using BERT model. Since the dataset is huge, around 50 thousand rows, I was thinking to use stratify sampling on dataset to reduce it to around 2-4 ...
1 vote
1 answer
371 views

Is there any concern for a pretrained model to overfitting to a fine-tuning task that has overlapping pretraining and training data?

Let's say my language model is pretrained on a general text corpus, and I want to use it for some specific downstream task that has it's datasets also included in the general corpus, is there any ...
0 votes
1 answer
372 views

How to write a generator to fine-tune transformer based models (Tensorflow)

I have been trying to write a generator for DistillBertFast model ...
2 votes
1 answer
492 views

Bertopic with embedding: unable to use find_topic

I've used BERTopic with success for the following tasks: get topics, visualise (topics, barcharts, documents ...) and DTM (extended to get area plot with considerable success). However, I am unable to ...
0 votes
0 answers
25 views

Approach for Multi-class Classification of texts

I'm trying to do a project where I have paragraphs and I need to classify them into multiple labels. The dataset is around 40k rows with labels. I understand there is no one right approach but should ...
0 votes
1 answer
98 views

Using BERT for the first time, what are the two columns of my test_results.tsv?

I followed the steps to feed in both dev, test, train.tsv to the model, trained it, then tried to classify test data, and I only have 1 feature, and the classification is binary, 1 or 0. I assumed my ...
0 votes
1 answer
210 views

Doubt in ELMO, BERT, Word2Vec

I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
1 vote
1 answer
79 views

How can I use contextual embeddings with BERT for sentiment analysis/classification

I have a BERT model which I want to use for sentiment analysis/classification. E.g. I have some tweets that need to get a POSITIVE,NEGATIVE or NEUTRAL label. I can't understand how contextual ...
0 votes
1 answer
2k views

Using KerasClassifier for training neural network

I created a simple neural network for binary spam/ham text classification using pretrained BERT transformer. The current pure-keras implementation works fine. I wanted however to plot certain metrics ...
2 votes
2 answers
344 views

Using BERT to extract a list of words and phrases from documents

I have a list of words and phrases (~3k items). What are my options to extract them from documents (~3M of job descriptions) with NLP? I do not have labeled data. For example my list of words and ...
0 votes
0 answers
64 views

job title normalizer

is there any way to normalize job titles using ml or nlp? examples: raw title: UX/UI Engineers normalized title: Software Engineers raw title: UX/UI Designer normalized title: Graphic Designers ...
1 vote
1 answer
307 views

Does high number of output labels affect the performance of BERT and how to handle the class imbalance issue while doing multi text classification?

I am using BERT to do multiclass text classification. The number of output classes I have to predict from is: 116 and there is high degree of class imbalance that I see. We have the following kind of ...
3 votes
2 answers
3k views

BERT - The purpose of summing token embedding, positional embedding and segment embedding

I read the implementation of BERT inputs processing (image below). My question is why the author chose to sum up three types of embedding (token embedding, positional embedding and segment embedding)?
1 vote
1 answer
35 views

Is it methodologically correct to use the data to be used for finetuning in the pretrain phase of the BERT model?

Let us assume the training of a BERT model. An initial pre-train is performed with a large data set A. Subsequently a finetuning is performed with a dataset B which is part of A, but now with labels ...
0 votes
1 answer
313 views

Loading a Model with weights and optimizers without creating an instance in PyTorch

I recently downloaded Camembert Model to fine-tune it for my purpose. Upon unzipping the file the contents are: Upon loading the model.pt file using pytorch: ...
0 votes
1 answer
142 views

Using BERT for co-reference resolving, what's the loss function?

I'm working my way around using BERT for co-reference resolving. I'm following this highly-cited paper BERT for Coreference Resolution: Baselines and Analysis (https://arxiv.org/pdf/1908.09091.pdf). I ...
0 votes
1 answer
62 views

Why the standard deviation of the BERT weight initialization is 0.02 by default

The purpose of weight initialization in the neural network is to keep the variance of calculation output in the layers to 1.0, and it depends on the calculations involved in the layers. Initializing ...
0 votes
2 answers
66 views

Bert model for document sentiment classification

I am trying to fine-tune a Bert model for sentiment analysis. Instead of one sentence, my inputs are documents (including several sentences) and I am not removing dots. I was wondering if is it okay ...
0 votes
0 answers
14 views

The using of golden dataset in Augmented SBERT Training

I use the training strategy of Augmented SBERT (Domain-Transfer). In the code example they use the golden-dataset (STSb) for the training evaluator. Here two code snippets of the example of sentence-...
1 vote
0 answers
35 views

Interpretation of Evaluation Values of Augmented SBERT Training with EmbeddingSimilarityEvaluator()

I train a BI-Encoder to get an Augmented SBERT and I get a final training result. How can I interpret the following output of the final training result? ...

1
2 3 4 5
8