Questions tagged [language-model]

Language models are used extensively in Natural Language Processing (NLP) and are probability distributions over a sequence of words or terms.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

What does logits in Casual Language Modeling represent?

I am reading the docs for transformers by hugging face and I see that the logits produced by casual language models are of the shape (batch_size, sequence_length, config.vocab_size). I also read the ...
0 votes
1 answer
10 views

Model for binary classification of links as "Article" or "Other"

I'm creating a web crawler which must: Fetch a web page. Parse all <a> tags with hrefs on the page. Classify the tags as either article (Meaning the link ...
0 votes
1 answer
20 views

Do I need training data in multiple languages for a multilingual transformer?

I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case. ...
  • 189
0 votes
1 answer
48 views

Loss on whole sequences in Causal Language Model

I'd like to know, from an implementation point of view, when training a Causal Transformer such as GPT-2, if making prediction on whole sequence at once and computing the loss on the whole sequence is ...
0 votes
0 answers
13 views

Arguments of OpenIE to extract fewer event triples

I'm new to NLP and I'm trying to using OpenIE to extract event triples from texts. I looked into its documents but quite don't understand its arguments. For example, ...
0 votes
0 answers
17 views

Does the Lempel-Ziv-Welch algorithm have theoretical or practical use as a language model?

If we encode a string using the LZW algorithm, we obtain a dictionary which maps strings of increasing length onto output symbols and a sequence of output symbols. Is the LZW algorithm useful (...
0 votes
0 answers
19 views

What is the minimal number of examples for BERT-like language model for the model to train a word

I have heard rumors of a particular count of positive examples that allowed the model to train a given word (or context of it - when talking about MLM) to be ~40. I am wondering though about the ...
  • 207
1 vote
1 answer
17 views

Which format is preferrable to publish book dataset (plain or preprocessed)?

When I decide to publish collection of book texts as a dataset, should I do some preprocessing first or should I publish "plain texts"? For example, https://huggingface.co/datasets/...
  • 113
0 votes
0 answers
23 views

Clarification on "predict the next character given the previous 100 characters"

I am studying Justin Johnson's lecture on RNNs Lecture recording: https://www.youtube.com/watch?v=dUzLD91Sj-o&list=PL5-TkQAfAZFbzxjBHtzdVCWE0Zbhomg7r&index=12&t=3177s One of the examples ...
  • 101
2 votes
1 answer
55 views

A multi label text classification problem

I'm looking to solve a multi label text classification problem but I don't really know how to formulate it correctly so I can look it up.. Here is my problem : Say I have the document ...
  • 35
0 votes
0 answers
7 views

Can Domain-Adaption improve the performance of Sentiment Analysis?

Does Domain Adaption have any effect of results in Sentiment Analysis? I am going to train a BERT language model based on some texts particularly in Health area, then I want to apply Opinion Mining on ...
3 votes
0 answers
37 views

Understanding Kneser-Ney Formula for implementation

I am trying to implement this formula in Python $$ \frac{\text{max}(c_{KN}(w^{i}_{i-n+1} - d), 0)}{c_{KN}(w^{i-1}_{i-n+1})} + \lambda(c_{KN}(w^{i-1}_{i-n+1})\mathbb{P}(c_{KN}(w_{i}|w^{i-1}_{i-n+2})$$ ...
  • 237
0 votes
0 answers
33 views

Transformer model comparison for binary sentiment classification

On two independent datasets, I am comparing XLNet and BERT models with binary sentiment classification tasks: the Twitter dataset, where sentences are short, and the IMDB review dataset, where ...
0 votes
1 answer
53 views

Sequence-to-Sequence Transformer for Neural machine translation

I am using the tutorial in Keras documentation here. I am new to deep learning. On a different dataset Menyo-20k dataset, of about 10071 total pairs, 7051 training ...
  • 17
1 vote
1 answer
131 views

Why not rule-based semantic role labelling?

I have recently found some interest in automatic semantic role labelling. Most introductory texts (e.g. Jurafsky and Martin, 2008) present approaches based on supervised machine learning, often using ...
1 vote
0 answers
10 views

Conceptually, how to deal with facts and time in GPT-3 and Language Models

When exploring text generation using various large language models, I frequently come across generated text which presents facts which are plain out wrong. I am not talking about fake news or bias, ...
1 vote
0 answers
68 views

Variable batch size for inputs of different length

We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
0 votes
0 answers
11 views

How can incrementality be measured in NLP?

As the title suggests, I am searching for a method to numerically evaluate the incrementality of a language model. Is there any way this can be achieved? For example, when a language model receives an ...
0 votes
0 answers
73 views

What does the term "seed lexicon" means?

I am reading a research paper (NLP) and found the phrase "seed lexicon". Could someone please explain it in detail? Edit : A sample paper Leveraging Affective Bidirectional Transformers for ...
0 votes
1 answer
145 views

how to improve my imbalanced data NLP model?

I want to classify a patient's health as a prediction probability and get the top 10 most ill patients in a hospital. I have patient's condition notes, medical notes, diagnoses notes, and lab notes ...
2 votes
2 answers
144 views

Best approach for text classification of phrases with little syntactic difference

So I have the task of classifying sentences based on their level of 'change talk' shown. Change talk is a psychology term used in counseling sessions to express how much the client wants to change ...
  • 73
2 votes
1 answer
25 views

Phrase/Token labeling

Looking for suggestions on how to define the following NLP problem and different ways in which it can be modeled to leverage machine learning. I believe there are multiple ways to model this problem. ...
1 vote
1 answer
89 views

For text classification, would a BoW or Word Embeddings based model ever be better than a Language Model?

I've done a bit of research, with this being the best as far as objectively measuring quality, but wanted to ask from a theoretical perspective if BoW-based models (e.g. using TF-IDF) or word ...
  • 111
1 vote
0 answers
37 views

NLP-Problem, language model BERT?

Right now I am in the process of deciding on my masters thesis topic. Right now I and my professor are thinking about the possibility of having a large dataset of product requirements given in a ...
1 vote
0 answers
51 views

Why we shift target(output) by one offset in language modelling

I have been working in sequence prediction tasks (very similar to language modelling) where I want to predict the next token(s)/item(s) given past sequence of tokens. I have always taken an approach ...
  • 296
0 votes
1 answer
28 views

Implementing a model for a language to another

I have a dataset of sentences of language X and Y (2 columns, for example, "abc def lang" ==> "xyz pqrt mno uages"). I want to have a output as a table that translates word by ...
  • 11
0 votes
1 answer
181 views

How to predict the sentiment of the entities form the tweet?

I have a JSON file (tweets.json) that contains tweets (sentences) along with the name of the author. Objective 1: Get the most frequent entities from the tweets. Objective 2: Find out the sentiment/...
1 vote
1 answer
59 views

Dealing with high frequency tokens during masked Language modelling?

Suppose I am working with a Masked Language Model to pre-train on a specific dataset. In that dataset, most sequences have a particular token of a high frequency ...
  • 167
0 votes
3 answers
131 views

Is this a tried alternative to word embedding for NLP?

I'm searching for research related to my idea, but apparently cannot articulate it well enough to the search engines to show me what's been published on this. My idea: in a deep learning context (text ...
2 votes
1 answer
516 views

A simple attention based text prediction model from scratch using pytorch

I first asked this question in codereview SE but a user recommended to post this here instead. I have created a simple self attention based text prediction model using pytorch. The attention formula ...
  • 281
0 votes
1 answer
138 views

Training Objective of language model for GPT3

On page 34 of OpenAI's GPT-3, there is a sentence demonstrating the limitation of objective function: Our current objective weights every token equally and lacks a notion of what is most important to ...
  • 105
1 vote
2 answers
345 views

Would there be any reason to pretrain BERT on specific texts?

So the official BERT English model is trained on Wikipedia and BookCurpos (source). Now, for example, let's say I want to use BERT for Movies tag recommendation. Is there any reason for me to pretrain ...
1 vote
1 answer
235 views

what's the motivation behind BERT masking 2 words in a sentence?

bert and the more recent t5 ablation study, agree that using a denoising objective always results in better downstream task performance compared to a language model where denoising == masked-lm == ...
  • 1,307
3 votes
1 answer
79 views

how to programmatically introduce grammatical errors in sentences

I've a set of sentences in English language. I'm exploring ways to create a dataset of sentences with grammatical errors programmatically. The following options has been tried out randomly - identify ...
  • 285
1 vote
1 answer
78 views

Inference order in BERT masking task

In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order? For example: The ...
0 votes
1 answer
1k views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
  • 117
0 votes
1 answer
89 views

Optimal input setup for character-level text classification RNN

I want to classify 500-character long text samples as to whether they look like natural language using a character-level RNN. I'm unsure as to the best way to feed the input to the RNN. Here are two ...
2 votes
1 answer
224 views

From where does BERT get the tokens it predicts?

When BERT is used for masked language modeling, it masks a token and then tries to predict it. What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression ...
1 vote
1 answer
2k views

What is the difference between GPT blocks and Transformer Decoder blocks?

I know GPT is a Transformer-based Neural Network, composed of several blocks. These blocks are based on the original Transformer's Decoder blocks, but are they exactly the same? In the original ...
  • 5,697
1 vote
1 answer
331 views

Can I fine-tune the BERT on a dissimilar/unrelated task?

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned: "During pre-training, the model is trained on unlabeled data over different pre-training tasks." I am not sure if I ...
  • 117
1 vote
0 answers
58 views

For an n-Gram model with n>2, do we need more context at end of each sentence?

Jurafsky's book says we need to add context to left and right of a sentence: Does this mean, for example, if we've a corpus of three sentences: ...
  • 123
4 votes
1 answer
227 views

In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens?

When learning Add-1 smoothing, I found that somehow we are adding 1 to each word in our vocabulary, but not considering start-of-sentence and end-of-sentence as two words in the vocabulary. Let me ...
  • 123
2 votes
2 answers
63 views

Evaluating Language Model on specific topic

I have finetuned a pretrained Language Model(GPT-2) on a custom dataset of mine. I would like a way of evaluating the ability of my model to generate sentences of a specific predefined topic, given in ...
  • 123
1 vote
2 answers
225 views

State-of-the-art Python packages that can evaluate language similarity

I am trying to evaluate the likelihood of generating a specific sentence out of a large set of sentences. To do this, I start from a simple approach: training a custom n-gram language model and ...
3 votes
1 answer
92 views

When using padding in sequence models, is Keras validation accuracy valid/ reliable?

I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have ...
1 vote
0 answers
31 views

Effect of discounting parameter on Language Model Perplexity

The general formula for absolute discounting for calculating language model probabilities subtracts a discounting parameter d from the count of the ngram before calculating the probabilities. The ...
  • 41
0 votes
1 answer
359 views

Why does English ELMo model give embeddings for non-English words?

Here's the code from my notebook: ...
  • 131
2 votes
1 answer
61 views

Generate text using user-supplied keywords

I've got a use case where I need to generate sentences based on a set of user supplied keywords. Here is an example of what I need: User input: End-User: Data Scientists Region: Middle East ...
9 votes
2 answers
2k views

Is BERT a language model?

Is BERT a language model in the sense of a function that gets a sentence and returns a probability? I know its main usage is sentence embedding, but can it also provide this functionality?
1 vote
1 answer
80 views

Comparing Language Model of two corpora

I know using Conditional Language Model I can learn the probability of a sentence given the corpus I used to train my model. I will then be able to generate meaningful text by sampling from the ...
  • 71