Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

How should we cut the long text into different sections for GPT-2 ? Do we use [PAD] to ensure the complete sentence included?

For GPT-2 model (I believe same for BERT) we need to cut the long text into fixed length for pre-training. Just wondering the details of how this should be implemented. Do we need to cut the text ...
0
votes
0answers
8 views

Difference between using BERT as a 'feature extractor' and fine tuning BERT with its layers fixed

I understand that there are two ways of leveraging BERT for some NLP classification task: BERT might perform ‘feature extraction’ and its output is input further to another (classification) model ...
0
votes
0answers
10 views

How to detokenize a BertTokenizer output?

For example, let's tokenize a sentece "why isn't Alex' text tokenizing": ...
0
votes
0answers
14 views

How to find out what each of the layers in NN does?

I have a very simplified view of Neural Networks - you give it input and expected output, and all the rest is a black box. Is it possible to find out, especially in language models, what each layer ...
0
votes
0answers
15 views

Convert a pandas df with text to embedding matrix efficiently

What is the most efficient way of converting a pandas data frame with approx 5 000 000 rows, where there exists short texts in one of the columns to an embedding matrix? I am using the HuggingFace ...
1
vote
0answers
21 views

Remove subwords from BERT output

I'm trying to build a multilingual WSD system with BERT on top as the embedding layer. In order to have better performances, after BERT finishes its job (and performs Transfer Learning), I need to ...
1
vote
3answers
36 views

Fastest way for 1 vs all lookup on embeddings

I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array. I wish to compare a new unseen text to all the 1 ...
0
votes
0answers
17 views

Flair Custom NER

I'm working on a problem in the domain of NER. I have a dataset wherein I need to have custom tags for different entities. I don't know how to start or even where to start. I know that there are ...
2
votes
0answers
40 views

What should be the labels for subword tokens in BERT for NER task?

For any NER task, we need a sequence of words and their corresponding labels. To extract features for these words from BERT, they need to be tokenized into subwords. For example, the word ...
0
votes
3answers
74 views

Generating synonyms or similar words from multiples word embeddings

I am looking for a way to generate synonyms, using word embeddings. From one word, and from multiple words. Such as the two example below: "word" -> Word embedding -> generate synonym of "word" "...
0
votes
0answers
7 views

Calculating semantic coherence in a given speech transcript

I am trying to calculate the semantic coherence in a given paragraph/transcript, ie. if somebody goes off track while talking about a thing or topic - more specifically describing a picture (the ...
2
votes
1answer
29 views

How to generate a sentence with exactly N words?

Thanks to GPT2 pretrained model now it is possible to generate meaningful sequence of words with or without prefix. However a sentence should end with a proper endings (.,!,?). I am just wondering how ...
2
votes
1answer
41 views

BERT in production

I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
1
vote
1answer
119 views

Can I fine-tune BERT, ELMO or XLnet for Seq2Seq neural machine translation?

I'm working on neural machine translator that translates English sentences to American sign language sentences(e.g below). I've a quite small dataset - around 1000 sentence pairs. I'm wondering if it ...
0
votes
1answer
117 views

Is it possible feed BERT to seq2seq encoder/decoder NMT (for low resource language)?

I'm working on NMT model which the input and the target sentences are from the same language (but the grammar differs). I'm planning to pre-train and use BERT since I'm working on small dataset and ...
2
votes
0answers
61 views

BertPunc (punctuation restoration with BERT)

I've found the script https://github.com/nkrnrnk/BertPunc for punctuation restoration. And I have one question about this method. I will briefly explain the logic of the author. One of four tokens is ...
1
vote
0answers
14 views

Multimodal end-to-end deep learning

I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains ...
2
votes
1answer
65 views

Semantic text simiarity using BERT

Given two sentences, I want to quantify the degree of similarity between the two text-based on Semantic similarity. Semantic Textual Similarity (STS) assesses the degree to which two sentences are ...
3
votes
2answers
89 views

What are the elements in a BERT word embedding?

As far as I understand, BERT is a word embedding that can be fine-tuned or used directly. With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one ...
0
votes
0answers
27 views

How could I make the bert friendly dataset?

I want to create the bert train,test dataset from the pandas dataframe ...
3
votes
1answer
43 views

BERT word embedings for finding word definition

Can BERT, GPT or other contextualised embedings be used for finding word definitions? What would be the most effective and not complicated approach for tackling a sample task as described below. Map ...
0
votes
0answers
58 views

Cross layer parameter sharing in ALBERT Model

I am reading the paper "ALBERT: LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS". ALBERT uses cross layer parameter sharing to improve the model performance. I don't understand how ...
1
vote
1answer
182 views

What is a 'hidden state' in BERT output?

I'm trying to understand the workings and output of BERT, and I'm wondering how/why each layer of BERT has a 'hidden state'. I understand what RNN's have a 'hidden state' that gets passed to each ...
1
vote
1answer
221 views

How does BERT and GPT-2 encoding deal with token such as <|startoftext|>, <s>

As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode ...
3
votes
1answer
220 views

What is purpose of the [CLS] token and why its encoding output is important?

I am reading this article on how to use BERT by Jay Alammar and I understand things up until: For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we ...
1
vote
1answer
48 views

Predicting Missing Word in Text

I know about BERT and other solutions when you masking some words and try to predict them. But let say I have a text: Transformer have taken the of Natural Processing by storm, transforming the ...
2
votes
1answer
69 views

Multilingual Bert sentence vector captures language used more than meaning - working as interned?

Playing around with BERT, I downloaded the Huggingface Multilingual Bert and entered three sentences, saving their sentence vectors (the embedding of [CLS]), then ...
2
votes
2answers
23 views

User profiling based on multiple posts

I currently have collected a dataset of different social media posts for each user with labels assigned to each user. I tried to use LSTM, and BERT for the text classification problem, So for each ...
1
vote
0answers
61 views

fine tune BERT in a small GPU

I want to to fine tune the BERT base model but the only accelerated HW I have access to is a couple of Quadro 600 GPUs which only pack 1GB RAM and 96 CUDA cores each. My question: is it even possible ...
0
votes
2answers
341 views

Why is the decoder not a part of BERT architecture?

I can't see how BERT makes predictions without using a decoder unit, which was a part of all models before it including transformers and standard RNNs. How are output predictions made in the BERT ...
1
vote
1answer
23 views

Measuring quality of answers from QnA systems

I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of ...
0
votes
1answer
116 views

Can BERT/ELMo be used (or retrained) to generate a text in both directions?

Text generation is perhaps one of the fun things to do with old NGram or new BERT/ELMo models. I am wondering can BERT be used to generate text from the end of a sentence, or better in both directions....
7
votes
1answer
103 views

What are the good parameter ranges for BERT hyperparameters while finetuning it on a very small dataset?

I need to finetune BERT model (from the huggingface repository) on a sentence classification task. However, my dataset is really small.I have 12K sentences and only 10% of them are from positive ...
0
votes
0answers
133 views

what are the steps to train BertForMaskedLM model on custom corpus and load it again and test it on new sentences?

what are the steps to train BertForMaskedLM model on custom corpus and load it again and test it on new sentences? I followed the instructions shared in BER github page to train a language model "...
0
votes
0answers
18 views

NL2SQL, for real industrial application, what strategy to locate the exact table?

The datasets like WikiSQL is that the table corresponding to question is given. But in real industrial application, we have 100+ tables for 1 new question. Thank you!
1
vote
1answer
531 views

How can I feed BERT to neural machine translation?

I am trying to feed the input and target sentences to an NMT model, I am trying to use BERT here, But I don't have any idea how ...
0
votes
2answers
1k views

How to load the pre-trained BERT model from local/colab directory?

Hi i downloaded the BERT pretrained model (https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip) from here and saved to a directory in gogole colab and in local . when i ...
1
vote
0answers
42 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
2
votes
1answer
66 views

where to store embeddings for similarity search?

I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
2
votes
1answer
244 views

NLP Transformers: How to get a fixed sentences embedding vectors size?

I'm loading a language model from torch hub (CamemBERT a French RoBERTa-based model) and using it do embed some sentences: ...
0
votes
2answers
54 views

How to obtain the word vectors optimally

I have a list of strings as shown sent_list = ["Carrefour is in France", "Apple pie is delicious", "Amazon has just delivered", ...] My code to get word ...
2
votes
2answers
279 views

Similarity of words using BERTMODEL

I want to find the similarity of words using the BERT model within the NER task. I have my own dataset so, I don't want to use the pre-trained model. I do the following: ...
0
votes
0answers
20 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
1
vote
1answer
51 views

BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence

For example I have an original sentence. The word barking corresponds to the word that is missing. ...
2
votes
0answers
610 views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a W2V or Glove, ...
0
votes
1answer
176 views

BERT - How Question answering is different than classification

Basically I am trying to understand how question answering works in case of BERT. Code for both classes QuestionAnswering and Classification is pasted below for reference. My understanding is: <...
0
votes
1answer
43 views

TLDR Bot - Sentence Tagging w/ BERT

Currently making a bot that condenses news articles. I'm tagging sentences as important or not important using a simple BERT classifier. The results were... not great. I'm really interested in how I ...
0
votes
0answers
23 views

Best structure for a LSTM Bert sentence classifier

I'm interested in classifying sentences using BERT. Finetuning on a single sentence had very poor results. I'd like to add a forward and backward LSTM layer to try to improve results. I'm having ...
8
votes
4answers
4k views

How to get sentence embedding using BERT?

How to get sentence embedding using BERT? ...