Skip to main content

Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
0 votes
0 answers
25 views

Best model for enforcing corporate naming conventions

I'm working on a project (Python) to enforce the company naming convention of products on product lists provided by clients/suppliers. I'm having a list of company names (Standardised names) and those ...
Secret Ambush's user avatar
0 votes
0 answers
21 views

NLP model for word recovery (analogy to BERT, but letters)

I am working on solving the problem of restoring words in text where some letters are missing. For example (restore words where vowels are removed): Hll wrld -> Hello world n ltrntv ssssmnt sggsts -...
SoH's user avatar
  • 119
0 votes
0 answers
15 views

How can I make my Hugging Face fine-tuned model's config.json file reference a specific revision/commit from the original pretrained model?

I uploaded this model: https://huggingface.co/pamessina/CXRFE, which is a fine-tuned version of this model: https://huggingface.co/microsoft/BiomedVLP-CXR-BERT-specialized Unfortunately, CXR-BERT-...
Pablo Messina's user avatar
0 votes
0 answers
25 views

Fine-tuning pretrained model on 2 tasks with 2 labeled dataset

I am having difficulty using BERT for a sentiment analysis task that handles both aspect-based sentiment analysis (ABSA) and comment sentiment analysis. I know that using two separate classification ...
ndycuong's user avatar
0 votes
0 answers
14 views

How can self-attention be used to combine representations from long text?

The paper "How to Fine-Tune BERT for Text Classification?" discusses using self-attention to combine the representations of a long input text that has been broken into chunks (section 5.3.1)....
suse's user avatar
  • 3
3 votes
0 answers
50 views

Weird behaviour when using RobERTA for text classification

I have a dataset with around 70 classes and the dataset is largely balanced ~150 samples per class. I am finetuning RoBERTA-base for 4 epochs with a ...
user1274878's user avatar
2 votes
0 answers
37 views

Use text embeddings to map job descriptions to ESCO occupations

I'm trying to build a model to map job descriptions to ESCO occupations which is a taxonomy for job titles. Every ESCO occupations have a title, a description and some essential skills. Ideally I ...
GanaelD's user avatar
  • 21
0 votes
0 answers
28 views

Does Fine Tuning with Custom Label Build Upon the Capability of Zero Shot Classification or Does It Train from Scratch?

The task is to classify email text bodies into exclusive categories like feedback, complaint etc. I have a labelled dataset available having about 350 samples. I have tried the ...
Della's user avatar
  • 335
1 vote
1 answer
56 views

Reducing emails token count preprocessing for Large Email Datasets - Feeding LLMs

I have a large email dataset in .txt format and want to feed LLMs (like Gemini and ChatGPT) to provide answers based on email content. The token count for my email data is very high (~1M for 1K emails)...
Rafael Borja's user avatar
0 votes
0 answers
7 views

Subsequence classification

Given multiple paragraphs, is it possible to classify an entire paragraph while taking into account the surrounding paragraphs? Paragraph1 Paragraph2 Paragraph3 ...
BPDev's user avatar
  • 101
0 votes
0 answers
9 views

Will hypermeters tuned on sampled dataset work for the whole dataset?

I'm doing multi-label classification on text data using BERT model. Since the dataset is huge, around 50 thousand rows, I was thinking to use stratify sampling on dataset to reduce it to around 2-4 ...
Shaurya Uniyal's user avatar
0 votes
0 answers
26 views

Approach for Multi-class Classification of texts

I'm trying to do a project where I have paragraphs and I need to classify them into multiple labels. The dataset is around 40k rows with labels. I understand there is no one right approach but should ...
Shaurya Uniyal's user avatar
1 vote
1 answer
125 views

How can I use contextual embeddings with BERT for sentiment analysis/classification

I have a BERT model which I want to use for sentiment analysis/classification. E.g. I have some tweets that need to get a POSITIVE,NEGATIVE or NEUTRAL label. I can't understand how contextual ...
average_discrete_math_enjoyer's user avatar
0 votes
0 answers
77 views

job title normalizer

is there any way to normalize job titles using ml or nlp? examples: raw title: UX/UI Engineers normalized title: Software Engineers raw title: UX/UI Designer normalized title: Graphic Designers ...
pycoder's user avatar
1 vote
1 answer
35 views

Is it methodologically correct to use the data to be used for finetuning in the pretrain phase of the BERT model?

Let us assume the training of a BERT model. An initial pre-train is performed with a large data set A. Subsequently a finetuning is performed with a dataset B which is part of A, but now with labels ...
Álvaro Loza's user avatar
0 votes
1 answer
71 views

Why the standard deviation of the BERT weight initialization is 0.02 by default

The purpose of weight initialization in the neural network is to keep the variance of calculation output in the layers to 1.0, and it depends on the calculations involved in the layers. Initializing ...
mon's user avatar
  • 761
0 votes
0 answers
14 views

The using of golden dataset in Augmented SBERT Training

I use the training strategy of Augmented SBERT (Domain-Transfer). In the code example they use the golden-dataset (STSb) for the training evaluator. Here two code snippets of the example of sentence-...
Christian01's user avatar
1 vote
0 answers
37 views

Interpretation of Evaluation Values of Augmented SBERT Training with EmbeddingSimilarityEvaluator()

I train a BI-Encoder to get an Augmented SBERT and I get a final training result. How can I interpret the following output of the final training result? ...
Christian01's user avatar
0 votes
0 answers
31 views

Explanation : Simpler models beat BERT base

I have been trying to train different models for a multi-class classification task of texts. My data set consists of rows of text and its label. The texts are short sentences. I tried the following ...
eya_bklt's user avatar
  • 101
0 votes
0 answers
37 views

Assign layers and weights in BERT

I print the weight names and shape of the BERT transformer. Now, I want to assign the printed weight to the layers in the transformers architecture: In the following, I can assign query, key and ...
Christian01's user avatar
0 votes
0 answers
79 views

What happens when I set is_decoder to True in the bert API from huggingface?

Please help me understand the implications of initialising the bert model from huggingface with is_decoder parameter set to True ...
figs_and_nuts's user avatar
2 votes
1 answer
106 views

How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder

Bert has two types of tasks that it uses to learn contextual word embeddings: Masked word prediction Next sentence prediction I have read the paper and even there the training details are a little ...
figs_and_nuts's user avatar
0 votes
0 answers
48 views

F1 and Exact-Match (EM) Score in Extractive QA NLP

I have a question as to how the F1 should be calculated in NLP and whether the text normalization is optional or not. So I have been working on a project where we created a closed-domain extractive QA ...
tt40kiwi's user avatar
  • 101
0 votes
0 answers
26 views

TFRobertaSequenceClassification for Address Normalization task

I have dataset with two column: one with faulty addresses, and other with correct addresses. I want to train a model such that, I can use it later for correcting all the incoming faulty addresses. I ...
learner_account's user avatar
0 votes
0 answers
43 views

How can I avoid the irrelevant number of sentences in the result?

The nature of the data I have is not arranged; however, I'm trying to extract the appropriate sentences for each query as a sample for ground truth. Also, the most critical problem is that I use the ...
Begnnier's user avatar
0 votes
0 answers
68 views

How to use Bertweet model for topic modeling

The problem is implementation of Bertweet in a topic-modeling project with understandable output like BERTopic, i want to use it on a relatively large (20k tweets) unlabelled dataset to segment it ...
Rossin's user avatar
  • 1
1 vote
2 answers
167 views

Training model using BERT

I have generated dataset using chat gpt. Dataset has 9000 data recodes. It's 6 class sentiment analysis. classes are 0,1,2,3,4,5 I used 3000 recodes for training, 1200 recods for validation and ...
Sandun Tharaka's user avatar
0 votes
0 answers
95 views

Help understanding working of KeyBERT for keyphrase extraction

I'm fairly new to reading and understanding research papers, so I wanted to get a second opinion on whether my understanding of KeyBERT was correct. Here is a high level overview of my understanding ...
Prithvi's user avatar
  • 33
0 votes
0 answers
67 views

Combining Textual, Categorical and Numerical data for Semantic Search using SentenceTransformers model

I'm building a food semantic search model and I want to use a pre-trained SentenceTransformers model with cosine similarity. I'm using Epicurious dataset for the corpus which consists of textual (&...
Alex's user avatar
  • 1
0 votes
0 answers
24 views

How to deal with short text data using NLP models?

Now I want to use my own domain data to train NLP model like BERT. The following is the details of my data: data length distribution: over 70% of my data has the length shorter than 5 and the largest ...
Jackie Shi's user avatar
0 votes
0 answers
449 views

Adapting a BERT-based model from HuggingFace for NER (named entity recognition) and RE (relation extraction)?

Context: NER (named entity recognition) and RE (relation extraction) from sentences obtained from radiology reports (medical text). There is a BERT-based model from HuggingFace I would like to use for ...
Pablo Messina's user avatar
8 votes
3 answers
11k views

Why does everyone use BERT in research instead of LLAMA or GPT or PaLM, etc?

It could be that I'm misunderstanding the problems space and the iterations of LLAMA, GPT, and PaLM are all based on BERT like many language models are, but every time I see a new paper in improving ...
Ethan's user avatar
  • 183
0 votes
0 answers
52 views

Classification errors on 'bert-base-uncased' text classifier

Disclaimer : This is a long question, please be patient. Thanks in advance I am using bert-base-uncased for text-classification. I have 11 classes, and the classification is happening alright for most ...
Vinay Varahabhotla's user avatar
0 votes
0 answers
41 views

BERT + tensorflow + deterministic

Im using BERT in tensorflow, but when I try to turn it deterministic I got the error: "When determinism is enabled, random ops must have a seed specified. [[{{node dropout/dropout/random_uniform/...
Heloisa Rocha's user avatar
0 votes
0 answers
14 views

How to analyze social media data to see its impact on a game's sales

I work for a console gaming giant. We forecasted the sales for a RPG game that was to be released few months back. But the actual sales was twice the forecast. This compelled the developers to ...
Ritik P. Nayak's user avatar
0 votes
2 answers
72 views

building embeddings for Phrases from scratch

I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
Christina Valavani's user avatar
0 votes
1 answer
59 views

Word embeddings

I m looking into word embedding and I would like to ask if I could train words or sentences in two layers. And if I wanted that one layer is more important, how could I calculate it? For example ...
Christina Valavani's user avatar
0 votes
1 answer
109 views

BERT is a supervised learning or semi-supervised learning?

I use 'bert-base-cased' pre-trained model for encoding a dataset of text that was labeled to labels 0, 1. Then the encoded dataset is trained using BERT model imported from Transformer library. Does ...
Balive13's user avatar
0 votes
1 answer
462 views

Can bert uncased predict text classification on foreign data?

I am trying to do the fake news/real news classification and used a pre-trained bert uncased model as transfer learning and it gave a solid 81% accuracy. But the problem is while doing sanity checks, ...
Jasmin Wilson's user avatar
1 vote
1 answer
363 views

Building BERT tokenizer with custom data

I'm wondering if there is a way to train our own Bert tokenizer instead of using pre-trained tokenizer provided by huggingface?
Loius Leong's user avatar
0 votes
1 answer
246 views

what is the difference between NSP and text prediction

In BERT, NSP (Next Sentence Prediction) is for predicting next sentence based on context and Text prediction task is also for predicting next word or phrases. So, both are for predicting next sentence ...
tovijayak's user avatar
1 vote
0 answers
55 views

I can't get good performance from BERT

I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin I compare an LSTM model with a glove model and a BERT model. I ...
Seydou GORO's user avatar
1 vote
0 answers
146 views

Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding

I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
soph's user avatar
  • 113
0 votes
2 answers
66 views

detect/generate possible tokens for the dataset (name,type/category,signatures)

I have a dataset in the following format: name, type, signature Eg1 : A, 2, abc123 Eg2 : A, 2, ab3 Eg3 : A, 2, addc1 If we need to train the following dataset ...
jason's user avatar
  • 53
0 votes
2 answers
68 views

Bert model for document sentiment classification

I am trying to fine-tune a Bert model for sentiment analysis. Instead of one sentence, my inputs are documents (including several sentences) and I am not removing dots. I was wondering if is it okay ...
m sh's user avatar
  • 3
1 vote
2 answers
1k views

Fine-tuned MLM based RoBERTa not improving performance

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM. We took some sample data points (2M+) & fine-tuned ...
Kalsi's user avatar
  • 11
1 vote
1 answer
87 views

Below text-classification model gives accuracy of 0.77 only on one dataset and 0.99 on spam-ham dataset? What should I do to increase with my dataset?

...
rutvi's user avatar
  • 19
0 votes
1 answer
618 views

BERTopic: Is it okay to ignore the first two topics?

I used BERTopic to generate a topic model over a large dataset of texts. The result is very appealing and the modeled topics are mostly perfectly interpretable for a human, especially compared to ...
oberbus's user avatar
0 votes
1 answer
224 views

Doubt in ELMO, BERT, Word2Vec

I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
NeverGiveUp's user avatar
1 vote
2 answers
87 views

Fine-tune GPT on sketch data (stroke-3)

These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it. I was planning on using some pretrained models that are ...
ilved17's user avatar
  • 41

1
2 3 4 5
8