Questions tagged [language-model]

Language models are used extensively in Natural Language Processing (NLP) and are probability distributions over a sequence of words or terms.

Filter by
Sorted by
Tagged with
1
vote
1answer
113 views

Why not rule-based semantic role labelling?

I have recently found some interest in automatic semantic role labelling. Most introductory texts (e.g. Jurafsky and Martin, 2008) present approaches based on supervised machine learning, often using ...
1
vote
0answers
8 views

Conceptually, how to deal with facts and time in GPT-3 and Language Models

When exploring text generation using various large language models, I frequently come across generated text which presents facts which are plain out wrong. I am not talking about fake news or bias, ...
0
votes
0answers
15 views

Variable batch size for inputs of different length

We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
0
votes
0answers
10 views

How can incrementality be measured in NLP?

As the title suggests, I am searching for a method to numerically evaluate the incrementality of a language model. Is there any way this can be achieved? For example, when a language model receives an ...
0
votes
0answers
15 views

What does the term "seed lexicon" means?

I am reading a research paper (NLP) and found the phrase "seed lexicon". Could someone please explain it in detail? Edit : A sample paper Leveraging Affective Bidirectional Transformers for ...
0
votes
1answer
78 views

how to improve my imbalanced data NLP model?

I want to classify a patient's health as a prediction probability and get the top 10 most ill patients in a hospital. I have patient's condition notes, medical notes, diagnoses notes, and lab notes ...
2
votes
2answers
90 views

Best approach for text classification of phrases with little syntactic difference

So I have the task of classifying sentences based on their level of 'change talk' shown. Change talk is a psychology term used in counseling sessions to express how much the client wants to change ...
2
votes
1answer
16 views

Phrase/Token labeling

Looking for suggestions on how to define the following NLP problem and different ways in which it can be modeled to leverage machine learning. I believe there are multiple ways to model this problem. ...
1
vote
1answer
30 views

For text classification, would a BoW or Word Embeddings based model ever be better than a Language Model?

I've done a bit of research, with this being the best as far as objectively measuring quality, but wanted to ask from a theoretical perspective if BoW-based models (e.g. using TF-IDF) or word ...
1
vote
0answers
25 views

NLP-Problem, language model BERT?

Right now I am in the process of deciding on my masters thesis topic. Right now I and my professor are thinking about the possibility of having a large dataset of product requirements given in a ...
1
vote
0answers
24 views

Why we shift target(output) by one offset in language modelling

I have been working in sequence prediction tasks (very similar to language modelling) where I want to predict the next token(s)/item(s) given past sequence of tokens. I have always taken an approach ...
0
votes
1answer
26 views

Implementing a model for a language to another

I have a dataset of sentences of language X and Y (2 columns, for example, "abc def lang" ==> "xyz pqrt mno uages"). I want to have a output as a table that translates word by ...
0
votes
1answer
48 views

How to predict the sentiment of the entities form the tweet?

I have a JSON file (tweets.json) that contains tweets (sentences) along with the name of the author. Objective 1: Get the most frequent entities from the tweets. Objective 2: Find out the sentiment/...
1
vote
1answer
36 views

Dealing with high frequency tokens during masked Language modelling?

Suppose I am working with a Masked Language Model to pre-train on a specific dataset. In that dataset, most sequences have a particular token of a high frequency ...
0
votes
3answers
49 views

Is this a tried alternative to word embedding for NLP?

I'm searching for research related to my idea, but apparently cannot articulate it well enough to the search engines to show me what's been published on this. My idea: in a deep learning context (text ...
1
vote
1answer
173 views

A simple attention based text prediction model from scratch using pytorch

I first asked this question in codereview SE but a user recommended to post this here instead. I have created a simple self attention based text prediction model using pytorch. The attention formula ...
0
votes
1answer
46 views

Training Objective of language model for GPT3

On page 34 of OpenAI's GPT-3, there is a sentence demonstrating the limitation of objective function: Our current objective weights every token equally and lacks a notion of what is most important to ...
0
votes
0answers
18 views

Language Model with Attention not learning

Language model with attention layer is not learning after 20 epochs. Both training and validation loss increase together, while the accuracy flattens at around 7% The way input data is pipelined is by ...
0
votes
0answers
18 views

Multilingual alternatives for med7

I'm looking for alternatives for med7 library for other common languages. Training a custom NER model for different languages seems like not the right option to ...
1
vote
2answers
222 views

Would there be any reason to pretrain BERT on specific texts?

So the official BERT English model is trained on Wikipedia and BookCurpos (source). Now, for example, let's say I want to use BERT for Movies tag recommendation. Is there any reason for me to pretrain ...
1
vote
1answer
108 views

what's the motivation behind BERT masking 2 words in a sentence?

bert and the more recent t5 ablation study, agree that using a denoising objective always results in better downstream task performance compared to a language model where denoising == masked-lm == ...
3
votes
1answer
40 views

how to programmatically introduce grammatical errors in sentences

I've a set of sentences in English language. I'm exploring ways to create a dataset of sentences with grammatical errors programmatically. The following options has been tried out randomly - identify ...
1
vote
1answer
46 views

Inference order in BERT masking task

In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order? For example: The ...
0
votes
1answer
803 views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
0
votes
1answer
46 views

Optimal input setup for character-level text classification RNN

I want to classify 500-character long text samples as to whether they look like natural language using a character-level RNN. I'm unsure as to the best way to feed the input to the RNN. Here are two ...
1
vote
1answer
152 views

From where does BERT get the tokens it predicts?

When BERT is used for masked language modeling, it masks a token and then tries to predict it. What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression ...
1
vote
1answer
1k views

What is the difference between GPT blocks and Transformer Decoder blocks?

I know GPT is a Transformer-based Neural Network, composed of several blocks. These blocks are based on the original Transformer's Decoder blocks, but are they exactly the same? In the original ...
1
vote
1answer
254 views

Can I fine-tune the BERT on a dissimilar/unrelated task?

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned: "During pre-training, the model is trained on unlabeled data over different pre-training tasks." I am not sure if I ...
1
vote
0answers
54 views

For an n-Gram model with n>2, do we need more context at end of each sentence?

Jurafsky's book says we need to add context to left and right of a sentence: Does this mean, for example, if we've a corpus of three sentences: ...
3
votes
1answer
172 views

In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens?

When learning Add-1 smoothing, I found that somehow we are adding 1 to each word in our vocabulary, but not considering start-of-sentence and end-of-sentence as two words in the vocabulary. Let me ...
2
votes
2answers
57 views

Evaluating Language Model on specific topic

I have finetuned a pretrained Language Model(GPT-2) on a custom dataset of mine. I would like a way of evaluating the ability of my model to generate sentences of a specific predefined topic, given in ...
1
vote
2answers
177 views

State-of-the-art Python packages that can evaluate language similarity

I am trying to evaluate the likelihood of generating a specific sentence out of a large set of sentences. To do this, I start from a simple approach: training a custom n-gram language model and ...
1
vote
0answers
44 views

When using padding in sequence models, is Keras validation accuracy valid/ reliable?

I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
1
vote
0answers
19 views

Effect of discounting parameter on Language Model Perplexity

The general formula for absolute discounting for calculating language model probabilities subtracts a discounting parameter d from the count of the ngram before calculating the probabilities. The ...
0
votes
1answer
298 views

Why does English ELMo model give embeddings for non-English words?

Here's the code from my notebook: ...
2
votes
1answer
54 views

Generate text using user-supplied keywords

I've got a use case where I need to generate sentences based on a set of user supplied keywords. Here is an example of what I need: User input: End-User: Data Scientists Region: Middle East ...
9
votes
2answers
2k views

Is BERT a language model?

Is BERT a language model in the sense of a function that gets a sentence and returns a probability? I know its main usage is sentence embedding, but can it also provide this functionality?
1
vote
1answer
51 views

Comparing Language Model of two corpora

I know using Conditional Language Model I can learn the probability of a sentence given the corpus I used to train my model. I will then be able to generate meaningful text by sampling from the ...
1
vote
0answers
16 views

How to convert subword PPL to word level PPL?

I'm using this formula to covert subword perpexity to word perplexity: PPL_word = exp(log(PPL_subword) * num_subwords / num_words) The question is do I need to ...
0
votes
1answer
297 views

Feeding XLM-R embeddings to neural machine translation?

I’m very new to the field of deep learning. My aim is to make a translation between Catalan to Catalan Sign Language. The grammar of the two languages is different Input: He sells food. Output (...
3
votes
2answers
2k views

What are the elements in a BERT word embedding?

As far as I understand, BERT is a word embedding that can be fine-tuned or used directly. With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one ...
29
votes
4answers
21k views

What is purpose of the [CLS] token and why is its encoding output important?

I am reading this article on how to use BERT by Jay Alammar and I understand things up until: For sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we ...
2
votes
1answer
109 views

The differences between BNF and JSGF in NLP?

I wonder what the differences are between the BNF(Backus-Naur Form) and JSGF(Java Speech Grammar Format)? The former is a kind of context-free grammar taught in CS224, but I learned that the latter is ...
3
votes
0answers
151 views

Fine tune gpt2 via huggingface API for domain specific LM

i am using the script in the examples folder to fine-tune the LM for a bot meant to deal with insurance related queries. So if someone were to type "i am looking to modify my ..." , the autocomplete ...
0
votes
1answer
2k views

How to calculate perplexity in PyTorch?

I am wondering the calculation of perplexity of a language model which is based on ...
2
votes
3answers
2k views

Any good Implementations of Bi-LSTM bahdanau attention in Keras?

From past few weeks I'm trying to learn sequence to sequence machine translation modelling but I couldn't find any good examples/tutorials with bahdanau attention implemented. I did come across a ton ...
1
vote
1answer
101 views

Transfer learning between Language Model and classification

Following this fast.ai lecture, I am trying to understand the mechanism of Transfer Learning in NLP from a general Language Model (LM) to a classification problem. What is exactly taken from the ...
1
vote
1answer
237 views

BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence

For example I have an original sentence. The word barking corresponds to the word that is missing. ...
1
vote
1answer
226 views

Language modelling for Spell Checker

I am working on spell checkers, I want to create a spell checker, I am confused about which model to use Word-Level modelling Character-Level modelling plus I am preferring neural networks over ...
1
vote
0answers
26 views

Skip-gram trained on The Hobbit: no improvement in the similarity of the word representation

I've trained a simple skipgram NNLM (window size = 5) on The Hobbit. This is the rough pseudocode: ...