Questions tagged [bert]
BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers
357
questions
0
votes
0
answers
183
views
How do I improve the accuracy of a BERT-based multilabel text classification model?
I have a database table with 79,512 rows, each of which describes a category. Each row has a title and a description, and can even have a supercategory. Often, supercategories have categories.
I'm ...
- 149
0
votes
1
answer
252
views
Combining text and image features with different scales
I have computed text features using [SBERT][1] and image features using VGG-16. The text features range from -1.58 to 1.58, whereas the image features range between 0 and 521. I would want to ...
- 1
0
votes
1
answer
102
views
Sentiment analysis BERT vs Model from scratch
I am working on building a sentiment analyzer, the data I would like to analyze is social media data from twitter, once I have created a the model I want to integrate it into a simply webpage.
I have ...
0
votes
2
answers
161
views
how can I translate Whisper encodings to SBERT embeddings?
I'm using the Whisper model to recognize speech, and then matching the output text against a list of known questions by generating SBERT embeddings from the text and ranking the known questions by ...
- 103
0
votes
1
answer
89
views
why some authors said that BERT cannot be used for text prediction?
I was trying to get a grasp about BERT and found this post in DS StackExchange:
Can BERT do the next-word-predict task?
In broad terms, it says that Bert cannot be used for next-word prediction. I ...
- 217
0
votes
0
answers
47
views
FinBERT out of the box performance testing
I'm trying to perform an out of the box performance test for FinBERT using the financialphrasebank dataset(sentiment analysis) to get a baseline performance before I start finetuning the model. The ...
- 1
0
votes
0
answers
41
views
Pruning using BERTology
I am trying out some BERT based models for a question and answering task. I need models trained on squad v2.0. To cut down on the inference time , I'm trying out pruning. I came across the BERTology ...
- 103
0
votes
0
answers
121
views
cannot freeze RoBERTa model base layer
I want to Freeze my RoBERTa model base layer and only train on my classification layer, but i get the following error 'TFRobertaEmbeddings' object has no attribute 'parameters'. Here is my code
...
- 1
1
vote
1
answer
275
views
BERTopic Visualization
I new to topic modeling and I'm trying to use BERTopic inside of PyCharm. I'm struggling to ...
- 141
0
votes
0
answers
58
views
Predicting same tokens as base BERT model for token classification on custom dataset
I have a custom dataset with custom tag for each token in the text. I want to train a BERT model for classifying each token into its corresponding category. To do ...
user144851
0
votes
1
answer
147
views
What does Embeddings Array Represent in BERT's Feature Extraction?
I am new to academic NLP, and I had been tasked with to use BERT to extract features of a sentence.
...
- 23
0
votes
1
answer
72
views
How to Train Q&A model using Bert for multiple comma seperated values in a given data
I'm using the entire text book data by scraping the information of each chapter.
How do I highlight the spacy spancat NER or Bert Q&A based models to train multiple comma separated values in the ...
- 53
0
votes
2
answers
168
views
Should i remove french special characters and apostrophes
I am working on a french text preprocessing task, in order to prepare the data to train an NLP model. But I do not know if it is better to remove french special characters and apostrophes or keep them....
- 3
0
votes
0
answers
58
views
Best pre-trained model to use for french multi label classification
I am working on fine tuning an NLP model for multi label classification of french text.
I have tried fine tuning Camembert-base, and Bert-base. I made sure the classes are balanced. The dataset is ...
- 3
0
votes
3
answers
663
views
Combining sentence embeddings of two different models (sBERT and mBERT)
I am working on a chatbot that helps students.
So, I wanted to make use of bert model which has better performance on mathematics, which lead to me to math-bert, but the paper on it said that it was ...
- 23
0
votes
0
answers
20
views
BERT model improvement approach - general advice
I'm new to machine learning and trying to improve an nlp / BERT model. I'd like to know if there is a way to improve my results by inspecting the weights and attention of individual words and phrases.
...
- 101
2
votes
0
answers
1k
views
How many samples in dataset are required to fine-tune BERT for binary classification?
I'm trying to fine-tune a BERT-based model for a binary classification task (data is in English). The dataset I'm working with is quite small (~500 samples, out of which 80% are currently used for ...
- 21
0
votes
0
answers
25
views
Transfer learning (or fine-tuning) pre-trained model on non-text data (PyTorch)
I am currently fine-tuning a sentiment analysis bert-based model using PyTorch Trainer from hugging face. So far, so good.
I have easily managed to fine-tune the model on my text data. However, I'd ...
- 101
0
votes
0
answers
12
views
Should I standardise my domain specific text prior to annotation/creation of a Q&A dataset?
I'm going to create my own question and answer dataset to fine-tune a BERT model, however before starting I am trying to understand if any standardisation of the text needs to be performed.
The data I ...
- 1
0
votes
1
answer
432
views
Is there bias in matrix multiplications for self attention
When the query matrix Q is computed as $XW_Q$, ($W_Q$ is the weight matrix for the queries), is it implemented as a linear layer without bias? I see some blogs saying there is are bias terms as well.
...
- 13
-1
votes
1
answer
236
views
What are MLM and NSP models actually used for after they've been trained?
I am a Python programmer working with deep learning nets and I have recently built my own language models as well as I have fine-tuned popular models like BERT. MY question is - after these models ...
- 103
3
votes
2
answers
281
views
NER - What advantage does IO Format have over BIO Format
In this paper, the authors say that they used IO schema instead of BIO in their dataset, which, if I am not wrong, means they just tag the corresponding Entity Type or "O" in case the word ...
- 33
0
votes
0
answers
197
views
CNN-BERT Text Classification good results on train and val, but bad prediction on testing
I built a Keras model to predict hoax news and true news using the CNN-BERT Text Classification algorithm with Categorical Classification, with label 1 indicating a hoax and 0 indicating true news.
...
- 1
0
votes
0
answers
80
views
Word embedding for Non-NLP words
I would like to embed words with a context, but that is not "Natural Language" - but just a list of words about more or less the same topic. Is there a way to use this context for the ...
- 323
0
votes
1
answer
55
views
Ordering training text data by length
If I have text data where the length of documents greatly varies and I'd like to use it for training where I use batching, there is a great chance that long strings will be mixed with short strings ...
0
votes
1
answer
317
views
Do I need to train a tokenizer when training SBERT with MLM?
I have trained a SBERT model with MLM on my own corpus which is somewhat domain specific using these guides:
https://ireneli.eu/2021/03/28/deep-learning-19-training-mlm-on-any-pre-trained-bert-models/
...
- 153
1
vote
1
answer
892
views
How does BERT produce CLS token? Internally does it do max-pooling or avarage pooling?
I ran experiment to compare max-pooled word tokens vs CLS token for sentence classification and CLS clearly wins. Trying to understand how BERT generates CLS token embedding if its better than max or ...
- 113
1
vote
1
answer
804
views
Fine tuning BERT without pre-training it on domain specific corpus
I'm building an internal semantic search engine using BERT/SBERT + ElasticSearch 8 where answers are retrieved based on their cosine similarity with a query.
The documents to be searched are somewhat ...
- 153
1
vote
1
answer
36
views
Good NLP model for computationally cheap predictions that can reasonably approximate language model given large training data set
I have a corpus of about one billion sentences, in which I am attempting to resolve NER conflicts (when two terms overlap in a sentence). My initial plan is to have an SME label the correct tag in ...
- 133
3
votes
1
answer
58
views
"cross-validation on the training set" while development and test set are distinct from the training: does it make sense? semantic mistake?
I got stuck on this paragraph from the academic article "Measuring news sentiment": https://www.sciencedirect.com/science/article/pii/S0304407620303535#tbl3
"As is best practice, we ...
- 31
1
vote
1
answer
359
views
Bertopic with embedding: unable to use find_topic
I've used BERTopic with success for the following tasks: get topics, visualise (topics, barcharts, documents ...) and DTM (extended to get area plot with considerable success).
However, I am unable to ...
- 111
0
votes
1
answer
63
views
One word changes everything NLP
I have a classification model (BERT) that classifies sentences as either question or normal sentences. But whenever a sentence has "how" word, the model chooses "question" class.
...
- 91
1
vote
2
answers
650
views
Extend BERT or any transformer model using manual features
I have been doing a thesis in my citation classifications. I just implemented Bert model for the classification of citations. I have 4 output classes and I give an input sentence and my model returns ...
- 11
0
votes
1
answer
131
views
How to use a `lr_scheduler` when you don't known how many training steps to do?
I am trying to fine-tune a BERT model, but instead of doing it a fix number of training step, I want to use a stalling policy and allow it to run until the model stalls for N evaluations. However, I ...
- 101
0
votes
1
answer
284
views
Input length of Sentence BERT
Can Sentence Bert embed an entire paragraph instead of only a sentence? For example, a description of a movie.
If so, what if the word counts exceeded the input limit? I remember Bert can only take up ...
- 165
0
votes
1
answer
62
views
**tokens when tokens is a dictionary
Trying to understand the code from https://www.analyticsvidhya.com/blog/2021/05/measuring-text-similarity-using-bert/
I am looking at understanding the syntax on these two lines:
...
- 11
2
votes
1
answer
1k
views
why there is no preprocessing step for training BERT?
I would like to train a BERT model from scratch. I read the paper as well as a few online material. It seems there is no preprocessing involved. e.g. removing punctuation, stopwords ...
I wonder why ...
- 323
2
votes
2
answers
279
views
Usage of Word2Vec
Sorry for the basic doubt,
I would like to know if I can use my Word2Vec straight for classification without using LSTM. My assumption is it’s not possible because the ordering of the words will not ...
- 109
0
votes
1
answer
68
views
Transformers vs RNN basic doubt
I have a basic doubt. Kindly clarify this.
My doubt is, When we are using LSTM's, We pass the words sequentially and get some hidden representations.
Now transformers also does the same thing except ...
- 109
0
votes
0
answers
139
views
How can I build and train mode for Arabic word embedding from scratch using BERT and share the model on hugging face?
my project is (building an Arabic word embedding model). I want to build my own model on hugging face like (aubmindlab/AraBERT model) for Arabic language using Bert for word embedding.
How can I start ...
- 121
0
votes
1
answer
381
views
Limitations of NLP BERT model for sentiment analysis
I am reading a paper, where the authors assess online public sentiment in China in response tot the government's policies during Covid-19, using a Chinese BERT model. The author's objective is not ...
- 127
1
vote
1
answer
155
views
Why do RNN text generation models treat word prediction as a classification task?
In many of the sources I have found regarding text generation with word-based RNN models (LSTM or GRU), the model is trained to perform a classification task across the vocabulary (such as with ...
- 113
0
votes
1
answer
928
views
How to get all 3 labels' sentiment from finbert instead of the most likely label's?
I'm using bert to do sentiment analysis. I previous used cardiffnlp's twitter-roberta-base-sentiment, https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment.
It gives the the usage on its ...
- 165
0
votes
1
answer
234
views
What did Sentence-Bert return here?
I used sentence bert to embed sentences from this tutorial https://www.sbert.net/docs/pretrained_models.html
...
- 165
0
votes
1
answer
325
views
How to reduce the size of Bert model(checkpoint/model_state.bin) using pytorch
I used torch.quantization.quantize_dynamic to reduce the model size but it is reducing my prediction Accuracy score.
I'm using that model file inside the Flask and ...
0
votes
0
answers
37
views
Mapping of an unseen Field/word to an existing description (in the input data), given Field and their respective descriptions as input/training data
I am working on a NLP problem.
Problem Statement
Given the input of fields & Labels and the respective descriptions, the goal is to the map a new unseen field to one of the most appropriate ...
- 319
0
votes
0
answers
309
views
Using BERT embeddings as input for transformer architecture
I will use BERT's embedding weights (as discussed here) for embedding in embedding layers of the transformer model. But my question is: don't embeddings of BERT already go through the whole encoding ...
- 91
1
vote
0
answers
407
views
What are the inputs of encoder and decoder layers of transformer architecture?
In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
- 91
0
votes
0
answers
53
views
Parameters for training a sentence-similarity model using Bert?
I have a list of sentences :
sentences = ["Missing Plate", "Plate not found"]
I am trying to find the most similar sentences in the list by ...
1
vote
1
answer
279
views
Model to implement Question Answering System over structured data
I need to write a program(like a chatbot) that retrieves an answer from a CSV datafile based on a question user asks. So for example if the CSV stores list of products and its specifications in 5-10 ...
