Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Bert Client fails to start and give an error

I have been trying to use the BERT model by google on my local machine. I have installed the latest version on Python3 and Pip3 but when I try to start the client it throws an error. Here is the ...
0
votes
0answers
32 views

BERT or ELMo for Document Similarity

Does anyone use BERT or ELMo language models to determine the similarity between two text documents? My question aims to collect all possible ways for combining the contextual word embeddings ...
2
votes
0answers
31 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
0
votes
0answers
21 views

How can I add custom numerical features for training to BERT fine tuning?

I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy. My original dataset consists of a pair of sentences (like MRPC ...
2
votes
0answers
54 views

How to add a CNN layer on top of BERT?

I am just playing with bert (Bidirectional Encoder Representation from Transformer) Research Paper Suppose I want to add any other model or layers like Convolutional Neural Network layers (CNN), Non ...
0
votes
0answers
27 views

Text classifiaction for large datasets using Transfer learning

I am trying to do text classification on a very large set of documents using the pretrained GPT model. The problem is GPT takes max sequence length $\le$ 1024. I can't truncate the data as I need to ...
0
votes
0answers
17 views

Finetuning BERT

Referring to the PyTorch port by huggingface of the native BERT library, I want to fine-tune the generated model on my personal dataset containing raw text. Could you please point out how this can be ...
1
vote
0answers
42 views

BERT vs Word2VEC: Is bert disambuguate the meaning of the word vector?

Word2vec: Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are ...
1
vote
0answers
8 views

bert_large is inaccurate compared to bert_base

I have posted this question on github official site too - Issue 708 I have been using bert_Base for Question and answering. Mostly it is good, but for production grade it needs to answer more ...
1
vote
1answer
61 views

Calculating cosine similarity between 3D arrays using Python

I have two matrices with multiple columns and three rows each. I calculated the cosine similarity (sklearn) but it gives the result as a matrix. How can I obtain one single value? The matrices are the ...
2
votes
1answer
253 views

What is whole word masking in the recent BERT model?

I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it: In the original pre-...
1
vote
1answer
35 views

Get long answers from BERT

We are using Google BERT for question and answering. We are using vanialla bert-base-uncased as well as squad trained checkpoints. The answers from BERT are very short and crisp. For example, if ...
2
votes
1answer
36 views

How to classify neutral sentiments using BERT

We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also? https://colab.research.google.com/github/google-...
0
votes
1answer
25 views

Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?

I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or ...
0
votes
0answers
19 views

Best service to host fine-tuned bert model?

I want to offer my fine-tuned bert model over the cloud. Are there any easy services to do this while only paying for usage (instead of an entire server)?
2
votes
0answers
86 views

BERT: it is possible to use it for topic modeling?

I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
2
votes
1answer
76 views

Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector

In the official github page of BERT, it mentions that: In certain cases, rather than fine-tuning the entire pre-trained model end-to-end, it can be beneficial to obtained pre-trained contextual ...
2
votes
0answers
32 views

meaning of fine-tuning in nlp task

There are two types of transfer learning model. One is feature extraction, where the weights of the pre-trained model are not changed while training on the actual task and other is the weights of the ...
1
vote
1answer
123 views

what is the first input to the decoder in a transformer model?

The image is from url: Jay Alammar on transformers K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each ...
2
votes
1answer
190 views

bert-as-service maximum sequence length

I installed bert-as-service (bert-as-service github repo) and tried encoding some sentences in Japanese on the multi_cased_L-12_H-768_A-12 model. It seems to work ...
0
votes
1answer
69 views

What is the use of [SEP] in paper BERT?

I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun. However, I have a question. If I have 2 sentences, which are s1 and s2, and our fine-tuning ...
0
votes
0answers
29 views

Which is the “most properly working” Bert-Ner repository

I am trying to find a repository in Github to get a Pytorch-reimplementation of the Bert model for NER task. So far, I found the following repos: https://github.com/kamalkraj/BERT-NER https://github....
12
votes
2answers
2k views

What is GELU activation?

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as $$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which appriximates to $$0.5x(1 + tanh[\sqrt{ 2/π}(x + 0.044715x^...
0
votes
0answers
44 views

BERT has a non deterministic behaviour

I am using the BERT implementation in https://github.com/google-research/bert for feature extracting and I have noticed a weird behaviour which I was not expecting: if I execute the program twice on ...
1
vote
0answers
81 views

Output range of BERT model shrinks after fine-tuning on domain specific dataset

My model's sigmoid output range has shrunk after transfer learning with small a dataset. My pretrained model has an output range of 0 to 1. After fine-tuning with a smaller domain specific dataset, ...
3
votes
0answers
957 views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
1
vote
0answers
77 views

Paragraph Generator using BERT or GPT

I am trying to generate similar sentences, called paragraph generation. For example, what is the name of eldest brother of ram? - For this paragraphs can be - who is oldest brother of ram? , Who is ...
2
votes
1answer
308 views

Incrementally Train BERT with minimum QnA records - to get improved results

We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/...
1
vote
0answers
116 views

How Transformer is Bidirectional - Machine Learning

Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/...
4
votes
1answer
738 views

Bert Fine Tuning with additional features

I want to use Bert for an nlp task. But I also have additional features that I would like to include. From what I have seen, with fine tuning, one only changes the labels and retrains the ...
7
votes
1answer
1k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?