Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
1
vote
1answer
31 views

How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>

As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode ...
2
votes
1answer
21 views

What is purpose of the [CLS] token and why its encoding output is important?

I am reading this article on how to use BERT by Jay Alammar and I understand things up until: For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we ...
1
vote
1answer
27 views

Predicting Missing Word in Text

I know about BERT and other solutions when you masking some words and try to predict them. But let say I have a text: Transformer have taken the of Natural Processing by storm, transforming the ...
2
votes
1answer
27 views

Multilingual Bert sentence vector captures language used more than meaning - working as interned?

Playing around with BERT, I downloaded the Huggingface Multilingual Bert and entered three sentences, saving their sentence vectors (the embedding of [CLS]), then ...
2
votes
2answers
17 views

User profiling based on multiple posts

I currently have collected a dataset of different social media posts for each user with labels assigned to each user. I tried to use LSTM, and BERT for the text classification problem, So for each ...
1
vote
0answers
29 views

fine tune BERT in a small GPU

I want to to fine tune the BERT base model but the only accelerated HW I have access to is a couple of Quadro 600 GPUs which only pack 1GB RAM and 96 CUDA cores each. My question: is it even possible ...
0
votes
2answers
53 views

Why is the decoder not a part of BERT architecture?

I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT ...
1
vote
1answer
18 views

Measuring quality of answers from QnA systems

I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of ...
0
votes
1answer
40 views

Can BERT/ELMo be used (or retrained) to generate a text in both directions?

Text generation is perhaps one of the fun things to do with old NGram or new BERT/ELMo models. I am wondering can BERT be used to generate text from the end of a sentence, or better in both directions....
6
votes
0answers
51 views

What are the good parameter ranges for BERT hyperparameters while finetuning it on a very small dataset?

I need to finetune BERT model (from the huggingface repository) on a sentence classification task. However, my dataset is really small. I have 12K sentences and only 10% of them are from positive ...
0
votes
0answers
29 views

what are the steps to train BertForMaskedLM model on custom corpus and load it again and test it on new sentences?

what are the steps to train BertForMaskedLM model on custom corpus and load it again and test it on new sentences? I followed the instructions shared in BER github page to train a language model "...
0
votes
0answers
14 views

NL2SQL, for real industrial application, what strategy to locate the exact table?

The datasets like WikiSQL is that the table corresponding to question is given. But in real industrial application, we have 100+ tables for 1 new question. Thank you!
1
vote
1answer
107 views

How can I feed BERT to neural machine translation?

I am trying to feed the input and target sentences to an NMT model, I am trying to use BERT here, But I don't have any idea how ...
0
votes
1answer
157 views

How to load the pre-trained BERT model from local/colab directory?

Hi i downloaded the BERT pretrained model (https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip) from here and saved to a directory in gogole colab and in local . when i ...
0
votes
0answers
39 views

Input data for BERT model using Tensor flow implementation

i'm implementing BERT using Hugging face transformers method and tensorflow . in the tutorial they've published the input of data need to be in the form of tf.dataset method or InputFeature object. ...
0
votes
0answers
20 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
2
votes
1answer
36 views

where to store embeddings for similarity search?

I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
2
votes
1answer
116 views

NLP Transformers: How to get a fixed sentences embedding vectors size?

I'm loading a language model from torch hub (CamemBERT a French RoBERTa-based model) and using it do embed some sentences: ...
0
votes
2answers
36 views

How to obtain the word vectors optimally

I have a list of strings as shown sent_list = ["Carrefour is in France", "Apple pie is delicious", "Amazon has just delivered", ...] My code to get word ...
2
votes
2answers
42 views

Similarity of words using BERTMODEL

I want to find the similarity of words using the BERT model within the NER task. I have my own dataset so, I don't want to use the pre-trained model. I do the following: ...
0
votes
0answers
16 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
1
vote
1answer
20 views

BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence

For example I have an original sentence. The word barking corresponds to the word that is missing. ...
0
votes
0answers
136 views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a W2V or Glove, ...
0
votes
1answer
103 views

BERT - How Question answering is different than classification

Basically I am trying to understand how question answering works in case of BERT. Code for both classes QuestionAnswering and Classification is pasted below for reference. My understanding is: <...
0
votes
1answer
24 views

TLDR Bot - Sentence Tagging w/ BERT

Currently making a bot that condenses news articles. I'm tagging sentences as important or not important using a simple BERT classifier. The results were... not great. I'm really interested in how I ...
0
votes
0answers
18 views

Best structure for a LSTM Bert sentence classifier

I'm interested in classifying sentences using BERT. Finetuning on a single sentence had very poor results. I'd like to add a forward and backward LSTM layer to try to improve results. I'm having ...
4
votes
2answers
1k views

How to get sentence embedding using BERT?

How to get sentence embedding using BERT? ...
0
votes
0answers
16 views

What will happen if we replace the transformer of BERT to evolved transformer?

If we replace the official BERT's transformer to evolved transformer, do the change accelerate the inference speed without losing accuracy?
1
vote
0answers
13 views

Can BERT embeddings be used to reproduce the original content of the text?

From what I understand, BERT provides contextualized embeddings that are not deterministic the way Word2Vec embeddings (i.e. the word "Queen" doesn't always produce the same vector, it'll be different ...
0
votes
0answers
13 views

What's the best way to store BERT training data (input IDs)

The tricky thing about the input IDs is what they're varying in length for each data sample, so regular hdf5 may not be ideal. Since Bert is so popular I am wondering if there's an established way to ...
7
votes
1answer
173 views

Why is word prediction an obsession in Natural Language Processing?

I have heard how great BERT is at masked word prediction, i.e. predicting a missing word from a sentence. In a Medium post about BERT, it says: The basic task of a language model is to predict ...
0
votes
1answer
144 views

BERT training on two tasks: what is the order of tasks?

I read that BERT has been trained on two tasks: Masked Language Modeling and Next Sentence Prediction. I want to gain clarity how exactly it was done. Was it initially trained on Masked Language ...
0
votes
1answer
51 views

Smallest Possible Dataset for Text Classification using BERT

What are your experiences for appropriate dataset sizes for usual text classification tasks using a finetuned BERT such as sentiment analysis? ~100 examples ~1000 examples ... ~10000000 examples ...
0
votes
0answers
49 views

Modifying BERT sentence encodings

I'm using BERT to encode sentences. The sentences I'm encoding are quite similar, meaning they all belong to the same overall topic. Therefor, I am using another parameter for measuring similarity. ...
2
votes
0answers
38 views

Structuring a LSTM Layer

I'm trying to improve an NER Bert sequence tagger using LSTM layers in TensorFlow. I'm a bit unclear on the interface and how a LSTM layer should be set up. Currently, I'm taking in 3-5 sentences and ...
1
vote
1answer
61 views

BERT : text classification and feature extractionn

I have tried multi-label text classification with BERT. Here is the sample input: $15.00 hour, customer service, open to industries One of the labels is Billing_rate and prediction score looks ...
0
votes
0answers
44 views

Summarization of documents using BERT

I have a collection of various documents that are partitioned according to their global topics. For each of these topics, I want to generate a new document that would summarize all of the ...
0
votes
0answers
453 views

BERT or ELMo for Document Similarity

Does anyone use BERT or ELMo language models to determine the similarity between two text documents? My question aims to collect all possible ways for combining the contextual word embeddings ...
2
votes
0answers
67 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
1
vote
1answer
437 views

How can I add custom numerical features for training to BERT fine tuning?

I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy. My original dataset consists of a pair of sentences (like MRPC ...
4
votes
2answers
479 views

How to add a CNN layer on top of BERT?

I am just playing with bert (Bidirectional Encoder Representation from Transformer) Research Paper Suppose I want to add any other model or layers like Convolutional Neural Network layers (CNN), Non ...
0
votes
0answers
30 views

Text classifiaction for large datasets using Transfer learning

I am trying to do text classification on a very large set of documents using the pretrained GPT model. The problem is GPT takes max sequence length $\le$ 1024. I can't truncate the data as I need to ...
0
votes
0answers
65 views

Finetuning BERT

Referring to the PyTorch port by huggingface of the native BERT library, I want to fine-tune the generated model on my personal dataset containing raw text. Could you please point out how this can be ...
5
votes
0answers
2k views

BERT vs Word2VEC: Is bert disambuguate the meaning of the word vector?

Word2vec: Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are ...
1
vote
0answers
21 views

bert_large is inaccurate compared to bert_base

I have posted this question on github official site too - Issue 708 I have been using bert_Base for Question and answering. Mostly it is good, but for production grade it needs to answer more ...
2
votes
1answer
1k views

Calculating cosine similarity between 3D arrays using Python

I have two matrices with multiple columns and three rows each. I calculated the cosine similarity (sklearn) but it gives the result as a matrix. How can I obtain one single value? The matrices are the ...
4
votes
1answer
1k views

What is whole word masking in the recent BERT model?

I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it: In the original pre-...
1
vote
1answer
146 views

Get long answers from BERT

We are using Google BERT for question and answering. We are using vanialla bert-base-uncased as well as squad trained checkpoints. The answers from BERT are very short and crisp. For example, if ...
2
votes
1answer
191 views

How to classify neutral sentiments using BERT

We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also? https://colab.research.google.com/github/google-...