Questions tagged [language-model]

Language models are used extensively in Natural Language Processing (NLP) and are probability distributions over a sequence of words or terms.

Filter by
Sorted by
Tagged with
1
vote
0answers
21 views

NLP-Problem, language model BERT?

Right now I am in the process of deciding on my masters thesis topic. Right now I and my professor are thinking about the possibility of having a large dataset of product requirements given in a ...
1
vote
0answers
23 views

Why we shift target(output) by one offset in language modelling

I have been working in sequence prediction tasks (very similar to language modelling) where I want to predict the next token(s)/item(s) given past sequence of tokens. I have always taken an approach ...
0
votes
1answer
26 views

Implementing a model for a language to another

I have a dataset of sentences of language X and Y (2 columns, for example, "abc def lang" ==> "xyz pqrt mno uages"). I want to have a output as a table that translates word by ...
0
votes
1answer
24 views

How to predict the sentiment of the entities form the tweet?

I have a JSON file (tweets.json) that contains tweets (sentences) along with the name of the author. Objective 1: Get the most frequent entities from the tweets. Objective 2: Find out the sentiment/...
1
vote
1answer
26 views

Dealing with high frequency tokens during masked Language modelling?

Suppose I am working with a Masked Language Model to pre-train on a specific dataset. In that dataset, most sequences have a particular token of a high frequency ...
0
votes
2answers
34 views

Is this a tried alternative to word embedding for NLP?

I'm searching for research related to my idea, but apparently cannot articulate it well enough to the search engines to show me what's been published on this. My idea: in a deep learning context (text ...
1
vote
0answers
32 views

A simple attention based text prediction model from scratch using pytorch

I first asked this question in codereview SE but a user recommended to post this here instead. I have created a simple self attention based text prediction model using pytorch. The attention formula ...
0
votes
1answer
28 views

Training Objective of language model for GPT3

On page 34 of OpenAI's GPT-3, there is a sentence demonstrating the limitation of objective function: Our current objective weights every token equally and lacks a notion of what is most important to ...
0
votes
0answers
14 views

Language Model with Attention not learning

Language model with attention layer is not learning after 20 epochs. Both training and validation loss increase together, while the accuracy flattens at around 7% The way input data is pipelined is by ...
0
votes
0answers
15 views

Multilingual alternatives for med7

I'm looking for alternatives for med7 library for other common languages. Training a custom NER model for different languages seems like not the right option to ...
1
vote
2answers
122 views

Would there be any reason to pretrain BERT on specific texts?

So the official BERT English model is trained on Wikipedia and BookCurpos (source). Now, for example, let's say I want to use BERT for Movies tag recommendation. Is there any reason for me to pretrain ...
1
vote
1answer
46 views

what's the motivation behind BERT masking 2 words in a sentence?

bert and the more recent t5 ablation study, agree that using a denoising objective always results in better downstream task performance compared to a language model where denoising == masked-lm == ...
3
votes
1answer
27 views

how to programmatically introduce grammatical errors in sentences

I've a set of sentences in English language. I'm exploring ways to create a dataset of sentences with grammatical errors programmatically. The following options has been tried out randomly - identify ...
1
vote
1answer
34 views

Inference order in BERT masking task

In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order? For example: The ...
0
votes
1answer
495 views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
0
votes
0answers
15 views

Where does BERT fit in the Machine Learning Hierarchy?

I am a newbie in the machine learning world and I need guidance from the professionals. I am trying to make a hierarchy starting from machine learning, then to deep learning and to BERT. I have read ...
0
votes
0answers
21 views

Where does the evaluation speed advantage of Transformer-XL come from?

The Transformer-XL paper claims an advantage in evaluation speed 363x-1874x than that of a baseline Transformer model. However, I do not understand where this massive difference comes from. Although ...
0
votes
1answer
34 views

Optimal input setup for character-level text classification RNN

I want to classify 500-character long text samples as to whether they look like natural language using a character-level RNN. I'm unsure as to the best way to feed the input to the RNN. Here are two ...
0
votes
0answers
13 views

Help understanding input to biaxial network for generating music

I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
1
vote
1answer
96 views

From where does BERT get the tokens it predicts?

When BERT is used for masked language modeling, it masks a token and then tries to predict it. What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression ...
1
vote
1answer
587 views

What is the difference between GPT blocks and Transformer Decoder blocks?

I know GPT is a Transformer-based Neural Network, composed of several blocks. These blocks are based on the original Transformer's Decoder blocks, but are they exactly the same? In the original ...
1
vote
1answer
195 views

Can I fine-tune the BERT on a dissimilar/unrelated task?

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned: "During pre-training, the model is trained on unlabeled data over different pre-training tasks." I am not sure if I ...
1
vote
0answers
51 views

For an n-Gram model with n>2, do we need more context at end of each sentence?

Jurafsky's book says we need to add context to left and right of a sentence: Does this mean, for example, if we've a corpus of three sentences: ...
3
votes
1answer
143 views

In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens?

When learning Add-1 smoothing, I found that somehow we are adding 1 to each word in our vocabulary, but not considering start-of-sentence and end-of-sentence as two words in the vocabulary. Let me ...
2
votes
2answers
56 views

Evaluating Language Model on specific topic

I have finetuned a pretrained Language Model(GPT-2) on a custom dataset of mine. I would like a way of evaluating the ability of my model to generate sentences of a specific predefined topic, given in ...
0
votes
0answers
9 views

How to: Plot global mean precipitation data from the NDAA onto map regions?

Wondering how to/ if I can display amount of rainfall across regions (in env variable) on the map areas so that the points only pertain to separate languages and sized by rainsetPropn. Have used data ...
1
vote
2answers
121 views

State-of-the-art Python packages that can evaluate language similarity

I am trying to evaluate the likelihood of generating a specific sentence out of a large set of sentences. To do this, I start from a simple approach: training a custom n-gram language model and ...
1
vote
0answers
36 views

When using padding in sequence models, is Keras validation accuracy valid/ reliable?

I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
1
vote
0answers
18 views

Effect of discounting parameter on Language Model Perplexity

The general formula for absolute discounting for calculating language model probabilities subtracts a discounting parameter d from the count of the ngram before calculating the probabilities. The ...
0
votes
1answer
214 views

Why does English ELMo model give embeddings for non-English words?

Here's the code from my notebook: ...
2
votes
1answer
40 views

Generate text using user-supplied keywords

I've got a use case where I need to generate sentences based on a set of user supplied keywords. Here is an example of what I need: User input: End-User: Data Scientists Region: Middle East ...
8
votes
2answers
1k views

Is BERT a language model?

Is BERT a language model in the sense of a function that gets a sentence and returns a probability? I know its main usage is sentence embedding, but can it also provide this functionality?
1
vote
1answer
42 views

Comparing Language Model of two corpora

I know using Conditional Language Model I can learn the probability of a sentence given the corpus I used to train my model. I will then be able to generate meaningful text by sampling from the ...
1
vote
0answers
14 views

How to convert subword PPL to word level PPL?

I'm using this formula to covert subword perpexity to word perplexity: PPL_word = exp(log(PPL_subword) * num_subwords / num_words) The question is do I need to ...
0
votes
1answer
275 views

Feeding XLM-R embeddings to neural machine translation?

I’m very new to the field of deep learning. My aim is to make a translation between Catalan to Catalan Sign Language. The grammar of the two languages is different Input: He sells food. Output (...
3
votes
2answers
2k views

What are the elements in a BERT word embedding?

As far as I understand, BERT is a word embedding that can be fine-tuned or used directly. With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one ...
15
votes
4answers
14k views

What is purpose of the [CLS] token and why is its encoding output important?

I am reading this article on how to use BERT by Jay Alammar and I understand things up until: For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we ...
2
votes
0answers
85 views

The differences between BNF and JSGF in NLP?

I wonder what the differences are between the BNF(Backus-Naur Form) and JSGF(Java Speech Grammar Format)? The former is a kind of context-free grammar taught in CS224, but I learned that the latter is ...
3
votes
0answers
139 views

Fine tune gpt2 via huggingface API for domain specific LM

i am using the script in the examples folder to fine-tune the LM for a bot meant to deal with insurance related queries. So if someone were to type "i am looking to modify my ..." , the autocomplete ...
0
votes
1answer
2k views

How to calculate perplexity in PyTorch?

I am wondering the calculation of perplexity of a language model which is based on ...
2
votes
3answers
2k views

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
1
vote
1answer
82 views

Transfer learning between Language Model and classification

Following this fast.ai lecture, I am trying to understand the mechanism of Transfer Learning in NLP from a general Language Model (LM) to a classification problem. What is exactly taken from the ...
1
vote
1answer
209 views

BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence

For example I have an original sentence. The word barking corresponds to the word that is missing. ...
1
vote
1answer
164 views

Language modelling for Spell Checker

I am working on spell checkers, I want to create a spell checker, I am confused about which model to use Word-Level modelling Character-Level modelling plus I am preferring neural networks over ...
1
vote
0answers
21 views

Skip-gram trained on The Hobbit: no improvement in the similarity of the word representation

I've trained a simple skipgram NNLM (window size = 5) on The Hobbit. This is the rough pseudocode: ...
0
votes
1answer
213 views

How to feed data for ngram model?

I want to train an ngram language model Let's say I have the following corpus: ...
6
votes
1answer
5k views

What is whole word masking in the recent BERT model?

I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it: In the original pre-...
4
votes
1answer
244 views

How should I treat these non-English documents in the NLP task?

So I have a small corpus of about 30k documents and about 50 documents in this corpus are in other languages (Persian, Chinese, Arabic, German, Spanish etc). I will be using this corpus for training a ...
1
vote
1answer
170 views

Why Heaps' Law Equation looks so different in this NLP course?

I'm actually not sure if this question is allowed on this community since it's more of a linguistics question than it is a data science question. I've searched extensively on the Web and have failed ...
1
vote
0answers
42 views

Why is MLP working similar to RNN for text generation

I was trying to perform text generation using only a character level feed-forward neural network after having followed this tutorial which uses LSTM. I one-hot encoded the characters of my corpus ...