Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
0
votes
1answer
26 views

Smallest Possible Dataset for Text Classification using BERT

What are your experiences for appropriate dataset sizes for usual text classification tasks using a finetuned BERT such as sentiment analysis? ~100 examples ~1000 examples ... ~10000000 examples ...
0
votes
0answers
14 views

Modifying BERT sentence encodings

I'm using BERT to encode sentences. The sentences I'm encoding are quite similar, meaning they all belong to the same overall topic. Therefor, I am using another parameter for measuring similarity. ...
2
votes
0answers
33 views

Structuring a LSTM Layer

I'm trying to improve an NER Bert sequence tagger using LSTM layers in TensorFlow. I'm a bit unclear on the interface and how a LSTM layer should be set up. Currently, I'm taking in 3-5 sentences and ...
1
vote
1answer
33 views

BERT : text classification and feature extractionn

I have tried multi-label text classification with BERT. Here is the sample input: $15.00 hour, customer service, open to industries One of the labels is Billing_rate and prediction score looks ...
0
votes
0answers
31 views

Summarization of documents using BERT

I have a collection of various documents that are partitioned according to their global topics. For each of these topics, I want to generate a new document that would summarize all of the ...
0
votes
0answers
203 views

BERT or ELMo for Document Similarity

Does anyone use BERT or ELMo language models to determine the similarity between two text documents? My question aims to collect all possible ways for combining the contextual word embeddings ...
2
votes
0answers
36 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
1
vote
0answers
103 views

How can I add custom numerical features for training to BERT fine tuning?

I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy. My original dataset consists of a pair of sentences (like MRPC ...
2
votes
0answers
177 views

How to add a CNN layer on top of BERT?

I am just playing with bert (Bidirectional Encoder Representation from Transformer) Research Paper Suppose I want to add any other model or layers like Convolutional Neural Network layers (CNN), Non ...
0
votes
0answers
29 views

Text classifiaction for large datasets using Transfer learning

I am trying to do text classification on a very large set of documents using the pretrained GPT model. The problem is GPT takes max sequence length $\le$ 1024. I can't truncate the data as I need to ...
0
votes
0answers
34 views

Finetuning BERT

Referring to the PyTorch port by huggingface of the native BERT library, I want to fine-tune the generated model on my personal dataset containing raw text. Could you please point out how this can be ...
1
vote
0answers
259 views

BERT vs Word2VEC: Is bert disambuguate the meaning of the word vector?

Word2vec: Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are ...
1
vote
0answers
11 views

bert_large is inaccurate compared to bert_base

I have posted this question on github official site too - Issue 708 I have been using bert_Base for Question and answering. Mostly it is good, but for production grade it needs to answer more ...
1
vote
1answer
307 views

Calculating cosine similarity between 3D arrays using Python

I have two matrices with multiple columns and three rows each. I calculated the cosine similarity (sklearn) but it gives the result as a matrix. How can I obtain one single value? The matrices are the ...
3
votes
1answer
651 views

What is whole word masking in the recent BERT model?

I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it: In the original pre-...
1
vote
1answer
82 views

Get long answers from BERT

We are using Google BERT for question and answering. We are using vanialla bert-base-uncased as well as squad trained checkpoints. The answers from BERT are very short and crisp. For example, if ...
2
votes
1answer
84 views

How to classify neutral sentiments using BERT

We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also? https://colab.research.google.com/github/google-...
0
votes
1answer
30 views

Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?

I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or ...
0
votes
0answers
38 views

Best service to host fine-tuned bert model?

I want to offer my fine-tuned bert model over the cloud. Are there any easy services to do this while only paying for usage (instead of an entire server)?
3
votes
0answers
309 views

BERT: it is possible to use it for topic modeling?

I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
2
votes
1answer
215 views

Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector

In the official github page of BERT, it mentions that: In certain cases, rather than fine-tuning the entire pre-trained model end-to-end, it can be beneficial to obtained pre-trained contextual ...
2
votes
0answers
86 views

meaning of fine-tuning in nlp task

There are two types of transfer learning model. One is feature extraction, where the weights of the pre-trained model are not changed while training on the actual task and other is the weights of the ...
1
vote
1answer
411 views

what is the first input to the decoder in a transformer model?

The image is from url: Jay Alammar on transformers K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each ...
2
votes
1answer
467 views

bert-as-service maximum sequence length

I installed bert-as-service (bert-as-service github repo) and tried encoding some sentences in Japanese on the multi_cased_L-12_H-768_A-12 model. It seems to work ...
0
votes
1answer
150 views

What is the use of [SEP] in paper BERT?

I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun. However, I have a question. If I have 2 sentences, which are s1 and s2, and our fine-tuning ...
0
votes
1answer
77 views

Which is the “most properly working” Bert-Ner repository

I am trying to find a repository in Github to get a Pytorch-reimplementation of the Bert model for NER task. So far, I found the following repos: https://github.com/kamalkraj/BERT-NER https://github....
13
votes
2answers
3k views

What is GELU activation?

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as $$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which appriximates to $$0.5x(1 + tanh[\sqrt{ 2/π}(x + 0.044715x^...
1
vote
1answer
89 views

BERT has a non deterministic behaviour

I am using the BERT implementation in https://github.com/google-research/bert for feature extracting and I have noticed a weird behaviour which I was not expecting: if I execute the program twice on ...
1
vote
0answers
114 views

Output range of BERT model shrinks after fine-tuning on domain specific dataset

My model's sigmoid output range has shrunk after transfer learning with small a dataset. My pretrained model has an output range of 0 to 1. After fine-tuning with a smaller domain specific dataset, ...
3
votes
0answers
2k views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
1
vote
0answers
103 views

Paragraph Generator using BERT or GPT

I am trying to generate similar sentences, called paragraph generation. For example, what is the name of eldest brother of ram? - For this paragraphs can be - who is oldest brother of ram? , Who is ...
2
votes
1answer
489 views

Incrementally Train BERT with minimum QnA records - to get improved results

We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/...
1
vote
0answers
240 views

How Transformer is Bidirectional - Machine Learning

Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/...
4
votes
1answer
938 views

Bert Fine Tuning with additional features

I want to use Bert for an nlp task. But I also have additional features that I would like to include. From what I have seen, with fine tuning, one only changes the labels and retrains the ...
7
votes
1answer
2k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?